Review Sampling
Review sampling helps reviewers track progress and focus on a subset of data. This allows for efficient quality checks while reducing review time.
This feature is only available for span labeling, row labeling, document labeling, and span + document labeling projects.
Configuration
You can enable this feature during project creation or in project settings.
During project creation
In step 4, set the conflict resolution to No consensus.
Enable Set review sampling rate in the bottom right corner.
The default sampling rate is 10%, but you can adjust it as needed.
In project settings
In project settings, go to the Assignment tab.
Set the conflict resolution to No consensus.
Enable Set review sampling rate in the bottom right corner.
View sampling progress
Once sampling rate is enabled and labeling is complete, reviewers can track progress in the Progress tab of the Review extension. This shows how many samples need review and how many have been completed.
How the progress is calculated
Span labeling projects
Progress is based on the number of unique labeled spans in each document.
Example: If Document A has 8 labeled spans and Document B has 11 labeled spans, with a 50% sampling rate, the reviewer needs to review 4 spans from Document A and 6 spans from Document B.
Row labeling projects
Progress is based on the number of rows in each document.
Example: If Document A has 10 rows and Document B has 5 rows, with a 50% sampling rate, the reviewer needs to review 5 rows from Document A and 3 rows from Document B.
Document labeling projects
Progress is based on the total number of documents.
Example: If a project has 12 documents and a 50% sampling rate, the reviewer needs to review 6 documents.
Span + document labeling projects
Progress combines both span and document calculations.
Example: If a project has 12 documents, each with 8 labeled spans, and a 50% sampling rate, the reviewer needs to review 6 documents and at least 4 spans within each document.
Marking the project as complete
When the feature is enabled, the system checks whether the required sampling rate is met before allowing the project to be marked as complete.
If the requirement is met, a confirmation dialog appears and you can mark the project as complete.
If the requirement is not met, a warning shows how many items remain. You must complete the required reviews before marking the project as complete.
This ensures the review process meets the required quality threshold before completion.
Last updated