Polygons and Pixel

Agreement

Agreement for polygons and pixel segmentations is calculated based on the average intersection over union (IoU) between each of the Top Labels and the Majority Label. The Top Labels are the ~5 best reads on each case, based on the labeler’s trailing average accuracy.

• High agreement: Significant overlap between Top Labels and the Majority Label.
• Low agreement: Limited overlap between Top Labels and the Majority Label.

Here, you can see how the agreement score would be calculated for Case 12345 if it had these three Top Labels.

The overall agreement for this case would be the average of the individual agreement scores for each top read (0.7, 0.3, 0.2) - which, in this case, is 0.4.

For multi-class segmentation, agreement for each individual class is calculated by the methodology above and the average is taken across all classes to determine agreement of the case.

Difficulty

Difficulty for polygons and pixel segmentations is based on the average IoU between the Top Labels and the Correct Label.

• High difficulty: Limited overlap between Top Labels and Correct Label.
• Low difficulty: Significant overlap between Top Labels and Correct Label.

Difficulty: 1 - average IoU between each of the Top Labels and the Correct Label

Looking at this example, here’s how we’d calculate the difficulty score for Case 54321.

The difficulty here will be calculated by subtracting the average overlap of the top reads with the Correct Label from 1 (1- average (0.7, 0.3, 0.2)), which results in a difficulty of 0.6.

For multi-class segmentation, disagreement for each individual class is calculated by the methodology above and the average is taken across all classes to determine disagreement of the case.