Review Sampling

Overview

This feature is only available for Span, Row, and Document Labeling projects.

The Review Sampling feature helps Reviewers track their progress in reviewing work done by Labelers. By enabling this feature, Reviewers can focus on a manageable portion of the data, ensuring quality assurance while saving time.

Configuration

You can enable this feature during project creation or from the project settings.

Project Creation

  1. In Project Creation Wizard step 4, the assignment step, set the conflict resolution to "No consensus".

  2. Enable the “Set review sampling rate” toggle, located at the bottom right corner.

  3. The default value is set to 10%, but user can configure the sampling rate based on the needs.

Project Settings

  1. Project settings can be opened through:

    • Active projects page — Click on the three-dot icon of the project you wish to configure, then select “View project settings”.

    • Inside a project — Navigate to File > Settings.

  2. Open the Assignment tab and set the conflict resolution to "No consensus".

  3. Enable the “Set review sampling rate” toggle, located at the bottom right corner.

Viewing Sampling Progress

Once the feature is enabled and Labelers have completed their labeling work, Reviewers can refer to the Progress tab in the Review extension to track the number of review samples they need to review and how many samples they have already reviewed.

How the progress is calculated

  • For Span Labeling projects, the review sampling calculation will be based on the number of unique spans with labels applied for each document.

    • For example, a project contains of Document A and B.

    • If Document A has 8 labeled spans and Document B has 11 labeled spans, and the review sampling rate is set to 50%, the Reviewer needs to review 4 spans from Document A and 6 spans from Document B.

  • For Row Labeling projects, the review sampling calculation will be based on the number of rows for each document. For example:

    • For example, a project contains of Document A and B.

    • If Document A has 10 rows and Document B has 5 rows, and the review sampling rate is set to 50%, the Reviewer needs to review 5 rows from Document A and 3 rows from Document B.

  • For Document Labeling projects, the review sampling calculation will be based on the number of documents in the project. For example:

    • For example, if a project consists of 12 documents, and the review sampling rate is set to 50%, the Reviewer needs to review 6 documents.

Last updated