Qualified Reads

A Read is one user’s answer on a single case. Not every Read is considered a Qualified Read.

A Qualified Read must meet the following criteria:

1. Labeler must meet or exceed the minimum participation threshold: A labeler’s initial reads in a task are automatically discarded. This ensures that each labeler has an appropriate amount of experience in the task before their reads could be considered Qualified Reads. Generally, a labeler must see ~10 - 25 cases before their read can be considered.
2. Labeler must meet or exceed the minimum trailing average accuracy threshold: A labeler must meet or exceed a trailing average accuracy on cases with answers (gold standards and labeled cases) in order for their reads to be considered Qualified Reads. Trailing average accuracy (also known as Q-score) is the proportion of correct reads of the most recent reads completed. The required minimum accuracy varies by task.

Only Qualified Reads are aggregated to create the Majority Label. All other reads are ignored and discarded from all metrics shown in the Centaur Labs portal.