Classification

Majority Label

The Majority Label is the answer choice that the majority (50%+) of labelers have picked. In a scenario where the vote is split equally among answer choices, the Majority Label answer is selected randomly from the top answer choices.

Correct Label

For Gold Standard cases, the Correct Label is what was provided by you or consultants contracted by you.

For Labeled cases, the Correct Label is assigned one of two ways:

• They have a majority answer that reaches consensus. A case reaches consensus when there is a minimum number of qualified reads (e.g., 3) that reach a certain level of agreement with one another (e.g., 75%). This ensures sufficient labeler confidence in the chosen answer.
• They accumulate the maximum number of qualified reads (typically 10-15). These cases should likely be reviewed manually by your team, as labelers were not able to clearly determine a correct answer.