Classification
Agreement
Agreement for classification tasks is based on the number of qualified reads for the Majority Label answer versus the total number of qualified reads.
- High agreement: Most qualified reads are for the Majority Label answer.
- Low agreement: Most of the qualified reads are not for the Majority Label answer.
Agreement formula: Qualified Reads with the Majority Label divided by total Qualified Reads.
| Answer choice | # qualified reads for each choice |
|---|---|
| Answer A | 3 |
| Answer B (Correct Label) | 4 |
| Answer C (Majority Label) | 8 |
| Answer D | 1 |
In this Gold Standard case example, agreement would be the number of reads for the Majority Label (8) over the number of total qualified reads (16) = 0.5.
Difficulty
Difficulty for classification tasks is based on the number of qualified reads for the Correct Label answer versus the total number of qualified reads.
- High difficulty: Most qualified reads are not for the Correct Label answer.
- Low difficulty: Most qualified reads are for Correct Label answer.
Difficulty formula: Qualified Reads without the Correct Label divided by total Qualified Reads.
Using the above example, difficulty would be the number of reads for everything but the Correct Label (12) over the number of total qualified reads (16) = 0.75.
Updated 3 months ago