Labeling states

Labeling StateDefinition
ReadyAn unlabeled case that we are not labeling yet.
In ProgressAn unlabeled case that is actively being reviewed by labelers.
Gold StandardA case that was submitted with a label by the customer team or an admin.
LabeledA case that has been labeled by Centaur’s network of experts.
CanceledA case that has been removed from labeling.
FlaggedA case that labelers have flagged as incorrect, inappropriate, or irrelevant.

Task labeling stage. The states above describe individual cases. Separately, each task has a labeling stage that controls whether labeling is open at all:

  • Pending — labelers can read the instructions but cannot label yet.
  • Open — labeling is active.
  • Assignment only — labeling is active, but labelers only work on cases assigned to them.
  • Paused — labeling is on hold. Labelers can still see the instructions and their own performance.
  • Closed — labeling has finished.

Your project manager sets the stage, and can attach a note that labelers see alongside it, for example to say when labeling will open. If you bring your own labelers to a task, they can only label while the task is open or assignment only.

Learn more here.