Range Selection

Dashboard

For range-selection tasks (i.e., NER, time range selection), average IoU and agreement are shown.

Average IoU

Similar to the segmentation task IoU metric, average intersection-over-union, or IoU, for a task is the IoU between each case’s Majority Label and the Correct Label averaged across all of Gold Standard cases within the task where the Centaur labelers have reached consensus.

The IoU for two snippets of highlighted text or selected deci-seconds is the area of their overlap divided by their combined area. The more closely two sections overlap, the higher the IoU.

If the Majority Label and Correct Label each contain multiple snippets of highlighted text or selected deci-seconds, the IoU between the two is calculated the same way: the intersection between the union of the Majority Label snippets and the union of the Correct Label snippets, divided by the union of all Majority Label and Correct Label snippets. Also, if both labels contain no highlighted or selected snippets (i.e. both are No Findings), then the IoU is defined to be 1 (i.e. perfect match).

Average agreement

Read about the case-level definition of agreement.

Task-level agreement is calculated by averaging the agreement scores of all Gold Standard and Labeled cases in the task.

Charts and Tables

Confusion matrix

For range selection tasks, the confusion matrix shows if the number of findings per case match between the Centaur labelers’ answers and the Correct Labels on the Gold Standard cases where our labelers have reached a consensus.

Read about matrices for multi-class range selection here.

Label distribution

The label distribution bar graph shows the breakdown of Gold Standard cases by class versus Labeled cases. The label distribution graph can be used to assess whether the class breakdown is consistent with your expectations if known.

Labeling rate

The labeling rate chart shows how many total cases were labeled by a given day. This chart will only include dates where contests were running, as that’s when cases move to Labeled.

Per-class Metrics

For multi-class range selection tasks we provide per-class metrics. This includes either the average IoU for each label class.