Manage gold standards

Gold standards (cases with known answers) can be managed via the Centaur API. A typical usage pattern would be:

1. Create cases in a labeling task (using the /tasks/imports/public/v1/add endpoint)
2. Add gold standards to cases where correct answers are known (using the /cases/goldStandards/public/v1/set endpoint)

Your task is now ready to be reviewed and launched to the crowd!

📘

API Credentials

The examples in this guide require having API credentials set up. If you haven't done so already, please follow the instructions in API credentials and Initiate API session.

About Gold Standards

Gold Standards are cases where the answer is already known, which are used for training and scoring purposes. Learn more about creating effective gold standards or review the flow of labeling states.

Setting Gold Standards

In order to set gold standards you will need to know the correct answer and the identifying information for your case. In most cases, the origin (unique filepath) of your data can be used as the identifying information.

In cases where Centaur has performed additional processing on your data, the origin may have been slightly modified. In this case, you may need to use case_id and content_id instead. This information can be retrieved via the Results API (see "Get Case Results" or "Get Task Results"). These endpoints will return information about the specific assets within each case as well as the current correct label (if one has been set).

Classification Tasks

Gold standards for classification tasks may be set with a PATCH request to /cases/goldStandards/public/v1/set. This request may use case_id or unique origin as the identifier. If both case_id and origin are provided, the case_id will take precedence in identifying the case to update.

The answer should be provided as a list of correct answers, all of which must be valid answer choices within the task. A string value to associate with the case may optionally be provided using the notes field.

curl --location --request PATCH 'https://api.centaurlabs.com/cases/goldStandards/public/v1/set?task_id=TASK_ID' \
--header 'X-API-KEY: YOUR_API_KEY_HERE' \
--header 'Authorization: Bearer YOUR_TOKEN_HERE' \
--header 'Content-Type: application/json' \
--data '{  
  "gold_standards": [  
    {  
      "case_id": CASE_ID,  
      "answer": [  
        "CORRECT_ANSWER"  
      ],  
      "notes": "NOTES"  
    },  
    {  
      "origin": UNIQUE_ORIGIN,  
      "answer": [  
        "CORRECT_ANSWER", "CORRECT_ANSWER"  
      ]  
    }  
  ]  
}'

Segmentation Tasks

🚧

Multi-Frame Segmentation

When setting gold standards in multi-frame (series) segmentation tasks, labels for all frames within a case must be provided. The most straightforward way to adhere to this constraint is typically to set all gold standards within a single series (grouping) with a single request.

Gold standards for segmentation tasks may may be set with a PATCH request to /cases/goldStandards/public/v1/set. This request must use a combination of case_id and content_id or unique origin as the identifier. If both case_id and origin are provided, the case_id will take precedence in identifying the case to update.

The answer should be provided as a list of objects, each of which identifies the answer_class and list of applicable segmentations in the geometries field using WKT format for that specific asset. For each frame where gold standards are submitted, any answer_class omitted from the answer will be assumed to have no findings. To indicate no findings for any label class for a specific frame, the answer field should be provided as an empty array []. A string value to associate with the case may optionally be provided using the notes field.

curl --location --request PATCH 'https://api-staging.centaurlabs.com/cases/goldStandards/public/v1/set?task_id=TASK_ID' \
--header 'X-API-KEY: YOUR_API_KEY_HERE' \
--header 'Authorization: Bearer YOUR_TOKEN_HERE' \
--header 'Content-Type: application/json' \
--data '{
  "gold_standards": [  
    {
      "case_id": CASE_ID,  
      "content_id": CONTENT_ID,  
      "answer": [
        {
          "answer_class": "foo",
          "geometries": [
            "WKT_SHAPE"
          ]
        }
      ]  
    },  
    {  
      "case_id": UNIQUE_ORIGIN,  
      "answer": [
        {
          "answer_class": "foo",
          "geometries": [
            "WKT_SHAPE"
          ]
        },
        {
          "answer_class": "bar",
          "geometries": [
            "WKT_SHAPE"
          ]
        }
      ],
      "notes": "This indicates findings for both 'foo' and 'bar' answer classes on this frame."  
    },
    {
      "case_id": CASE_ID,  
      "content_id": CONTENT_ID,  
      "answer": [],
      "notes": "This indicates 'no findings' for all answer classes on this frame."
    }
  ]  
}'