Thresholding

Thresholding for Classification Tasks

Thresholding is a post-processing review strategy that can be used to correct under or over-classification. For example, if you have an underrepresented answer class and cases with this answer are generally being missed by our labelers, you can apply this review strategy to identify candidates for this answer class. Download the task results to begin this process.

Automatic Thresholding

The threshold is defined as the number of reads needed for the most chosen answer choice for a case to be considered “Labeled”. Our default threshold is the most chosen answer choice must have 3+ more reads than the second most chosen answer choice.

As we collect multiple reads per case, you may want to redefine the threshold for certain classes. For example, you can redefine the threshold for an underrepresented class: any case with at least one read for that answer class can be considered to have that label. Similarly, in the event of an overrepresented answer class, you could increase the threshold to 5+ votes for that class, rather than our default 3+, for the case to have that label.

Example: In this example, “corrupted” is a rare answer choice and difficult to detect. Therefore, you may want to change the threshold so that the answer is assigned to "corrupted", even if just receives one read for that class. Therefore, case 0003 would be labeled as “corrupted”, as all cases with at least 1 read for “corrupted” would be considered part of the “corrupted” class.

Case ID	Labeling State	Correct Answer	Agreement	Qualified Reads	Reads "true"	Reads "false"	Reads "corrupted"
0001	Labeled	['true']	1	8	8	0	0
0002	Labeled	['true']	0.5	12	6	6	0
0003	Labeled	['true']	0.8	5	4	0	1

Note: This is not an exact representation of the classification results download and has been created to explain the concept above.

Manual Thresholding

You can also assess cases one by one to find good candidates for additional cases in rarer classes. Using the results download, filter for cases with at least one read for the rare answer choice (even though this answer choice is not the Correct Label). Then, determine if these cases are good candidates for the rare and difficult to detect answer class.

This strategy can also be used for overrepresented classes. Using the results download, filter for cases with 3+ read differential in the overrepresented class. Then, determine if these cases are good candidates for the second highest answer choice (after the overrepresented class) or if the threshold for the overrepresented class should be higher than 3+ reads.