scripts.analyze_agreement module

analyze_agreement.py is a script that analyzes the agreement between two annotations of the same file. The script measures:

• Object counts: are they the same?
• Object assignment: given the least-squares mapping of objects onto each other, to what extent do they differ?

For an overview of command-line options, call:

analyze_agreement.py -h

Alignment algorithm

The script uses a greedy alignment procedure.

First, it computes for each (truth, prediction) symbol pair their recall, precision, and f-score over pixels that fall within the mask (bounding box overlap may be misleading, mainly for parallel beams).

Each predicted symbol is then aligned to the ground truth symbol with the highest f-score. If the symbol classes of a (truth, prediction) pair do not match, their score gets set to 0. (This can be turned off using the --no_strict_clsnames option.)

Next, the alignment is cleaned up: if multiple predictions are aligned to a single ground truth, the one with the highest f-score is chosen and the other predicted symbols are considered unaligned.

Computing the output f-score

Finally, we sum all the f-scores of (truth, prediction) symbol pairs in the alignment.

Ground truth symbols that are not aligned to any predicted object also contribute a zero to the overall f-score.

scripts.analyze_agreement.align_cropobjects(truth, prediction, fscore=None)

Aligns prediction CropObjects to truth.

Parameters:

• truth – A list of the ground truth CropObjects.
• prediction – A list of the predicted CropObjects.

Returns:

A list of (t, p) pairs of CropObject indices into the truth and prediction lists. There will be one pair for each predicted symbol.

scripts.analyze_agreement.bbox_intersection(origin, intersect)

Returns the coordinates of the origin bounding box that are intersected by the intersect bounding box.

>>> bounding_box = 10, 100, 30, 110
>>> other_bbox = 20, 100, 40, 105
>>> bbox_intersection(bounding_box, other_bbox)
(10, 0, 20, 5)
>>> bbox_intersection(other_bbox, bounding_box)
(0, 0, 10, 5)
>>> containing_bbox = 4, 55, 44, 115
>>> bbox_intersection(bounding_box, containing_bbox)
(0, 0, 20, 10)
>>> contained_bbox = 12, 102, 22, 108
>>> bbox_intersection(bounding_box, contained_bbox)
(2, 2, 12, 8)
>>> non_overlapping_bbox = 0, 0, 3, 3
>>> bbox_intersection(bounding_box, non_overlapping_bbox) is None
True

scripts.analyze_agreement.build_argument_parser()

scripts.analyze_agreement.cropobjects_rpf(truth, prediction)

Computes CropObject pixel-level metrics.

Parameters:

• truth – A list of the ground truth CropObjects.
• prediction – A list of the predicted CropObjects.

Returns:

Three matrices with shape (len(truth), len(prediction): recall, precision, and f-score for each truth/prediction CropObject pair. Truth cropobjects are rows, prediction columns.

scripts.analyze_agreement.main(args)

scripts.analyze_agreement.pixel_metrics(truth, prediction)

Computes the recall, precision and f-score for the prediction CropObject given the truth CropObject.

scripts.analyze_agreement.rpf_given_alignment(alignment, r, p, n_not_aligned=0, strict_clsnames=True, truths=None, predictions=None)