Agreement

Agreement for range selection (i.e., NER, time range selection) is calculated based on the average overlap (determined by the average intersection over union, IoU) between all of the Qualified Reads and the Majority Label.

• High agreement: Significant overlap between all Qualified Reads and the Majority Label.
• Low agreement: Limited overlap between all Qualified Reads and the Majority Label.

For multi-class range selection, agreement for each individual class is calculated by the methodology above and the average is taken across all classes to determine agreement of the case.

Difficulty

Difficulty for range selection (i.e., NER, time range selection) is based on the average overlap (determined by the average intersection over union, IoU) between all of the Qualified Reads and the Correct Label.

• High difficulty: Limited overlap between all Qualified Reads and Correct Label.
• Low difficulty: Significant overlap between all Qualified Reads and Correct Label.

Difficulty formula: 1 - average IoU between all of the Qualified Reads and the Correct Label

For multi-class range selection, difficulty for each individual class is calculated by the methodology above and the average is taken across all classes to determine difficulty of the case.