scripts.analyze_agreement module¶
analyze_agreement.py
is a script that analyzes the agreement between two
annotations of the same file. The script measures:
 Object counts: are they the same?
 Object assignment: given the leastsquares mapping of objects onto each other, to what extent do they differ?
For an overview of commandline options, call:
analyze_agreement.py h
Alignment algorithm¶
The script uses a greedy alignment procedure.
First, it computes for each (truth, prediction)
symbol pair
their recall, precision, and fscore over pixels that fall within
the mask (bounding box overlap may be misleading, mainly for
parallel beams).
Each predicted symbol is then aligned to the ground truth symbol
with the highest fscore. If the symbol classes of a (truth, prediction)
pair do not match, their score gets set to 0. (This can be turned
off using the no_strict_clsnames
option.)
Next, the alignment is cleaned up: if multiple predictions are aligned to a single ground truth, the one with the highest fscore is chosen and the other predicted symbols are considered unaligned.
Computing the output fscore¶
Finally, we sum all the fscores of (truth, prediction)
symbol pairs in the alignment.
Ground truth symbols that are not aligned to any predicted object also contribute a zero to the overall fscore.

scripts.analyze_agreement.
align_cropobjects
(truth, prediction, fscore=None)[source]¶ Aligns prediction CropObjects to truth.
Parameters:  truth – A list of the ground truth CropObjects.
 prediction – A list of the predicted CropObjects.
Returns: A list of (t, p) pairs of CropObject indices into the truth and prediction lists. There will be one pair for each predicted symbol.

scripts.analyze_agreement.
bbox_intersection
(origin, intersect)[source]¶ Returns the coordinates of the origin bounding box that are intersected by the intersect bounding box.
>>> bounding_box = 10, 100, 30, 110 >>> other_bbox = 20, 100, 40, 105 >>> bbox_intersection(bounding_box, other_bbox) (10, 0, 20, 5) >>> bbox_intersection(other_bbox, bounding_box) (0, 0, 10, 5) >>> containing_bbox = 4, 55, 44, 115 >>> bbox_intersection(bounding_box, containing_bbox) (0, 0, 20, 10) >>> contained_bbox = 12, 102, 22, 108 >>> bbox_intersection(bounding_box, contained_bbox) (2, 2, 12, 8) >>> non_overlapping_bbox = 0, 0, 3, 3 >>> bbox_intersection(bounding_box, non_overlapping_bbox) is None True

scripts.analyze_agreement.
cropobjects_rpf
(truth, prediction)[source]¶ Computes CropObject pixellevel metrics.
Parameters:  truth – A list of the ground truth CropObjects.
 prediction – A list of the predicted CropObjects.
Returns: Three matrices with shape
(len(truth), len(prediction)
: recall, precision, and fscore for each truth/prediction CropObject pair. Truth cropobjects are rows, prediction columns.