Range Selection

We support labeling ranges within text and video entities. Named Entity Recognition (NER) is range labeling on text assets, whereas Time Range Selection in range labeling on video assets.

NER

Majority Label

Individual qualified reads on a named entity recognition (NER) task are aggregated to create the Majority Label.

Similar to classification, NER cases consider all qualified reads. This means that all qualified reads on a case are included in the aggregation.

NER qualified reads are combined using:

• Aggregate overlap: Minimum percentage of qualified reads needed with a highlighted character for that character to be included in the final Majority Label

For example, if the required overlap percentage is 60% that means that at least 60% of all qualified reads need to highlight a character for it to be included in the final label. Imagine a case with 5 qualified reads.

You can see here that only the text highlighted by 3+ qualified reads (60%+ of 5 reads) makes it to the final label. Text highlighted by <3 reads will not be included.

In the case of multi-class NER, this process is performed independently for each of the classes present.

Correct Label

For Gold Standard cases, the Correct Label is what was provided by you or consultants contracted by you.

For Labeled cases, the Correct Label is the same as the Majority Label once it has received the minimum number of qualified reads (e.g., 3) that reach a certain level of agreement with one another (e.g., 75%).

Time Range Selection

Majority Label

Individual qualified reads on a time range selection task are aggregated to create the Majority Label.

Similar to classification, time range selection cases consider all qualified reads. This means that all qualified reads on a case are included in the aggregation.

Time range selection qualified reads are combined using:

• Aggregate overlap: Minimum percentage of qualified reads needed with a selected deci-second for that deci-second to be included in the final Majority Label

For example, if the required overlap percentage is 60% that means that at least 60% of all qualified reads need to select a deci-second for it to be included in the final label.

In the case of multi-class time range selection, this process is performed independently for each of the classes present.

Correct Label

For Gold Standard cases, the Correct Label is what was provided by you or consultants contracted by you.

For Labeled cases, the Correct Label is the same as the Majority Label once it has received the minimum number of qualified reads (e.g., 3) that reach a certain level of agreement with one another (e.g., 75%).