Review Sampling
Overview
This feature is only available for: Span, Row, Document and Span + Document Labeling projects.
The Review Sampling feature helps Reviewers track their progress in reviewing work done by Labelers. By enabling this feature, Reviewers can focus on a manageable portion of the data, ensuring quality assurance while saving time.
Configuration
You can enable this feature during project creation or from the project settings.
Project Creation
In Project Creation Wizard step 4, the assignment step, set the conflict resolution to "No consensus".
Enable the “Set review sampling rate” toggle, located in the bottom right corner.
The default value is set to 10%, but user can configure the sampling rate based on the needs.
Project Settings
Project settings can be opened through:
Active projects page — Click on the three-dot icon of the project you wish to configure, then select “View project settings”.
Inside a project — Navigate to File > Settings.
Open the Assignment tab and set the conflict resolution to "No consensus".
Enable the “Set review sampling rate” toggle, located in the bottom right corner.
Viewing sampling progress
Once the feature is enabled and Labelers have completed their labeling work, Reviewers can refer to the Progress tab in the Review extension to track the number of review samples they need to review and how many samples they have already reviewed.
How the progress is calculated
Span Labeling Projects
The progress calculation is based on the number of unique labeled spans in each document.
Example: If Document A has 8 labeled spans and Document B has 11 labeled spans, with a review sampling rate of 50%, the Reviewer will need to review 4 spans from Document A and 6 spans from Document B.
Row Labeling Projects
The progress calculation is based on the number of rows in each document.
Example: If Document A has 10 rows and Document B has 5 rows, with a review sampling rate of 50%, the Reviewer will need to review 5 rows from Document A and 3 rows from Document B.
Document Labeling Projects
The progress calculation is based on the total number of documents in the project.
Example: If a project has 12 documents, with a review sampling rate of 50%, the Reviewer will need to review 6 documents.
Span + Document Labeling Projects
The progress calculation combines both Span and Document Labeling.
Example: If a project has 12 documents, each with 8 labeled spans, and a review sampling rate of 50%, the Reviewer will need to review 6 documents and at least 4 spans within each document.
Marking the project as complete
When this feature is enabled, a validation process ensures the Reviewer has met the required review sampling rate before allowing the project to be marked as complete.
The validation process includes:
Review Sampling Rate Check
When a Reviewer attempts to mark the project as complete, the system checks if the required review sampling rate has been met.
Completion Confirmation
If the Reviewer has reviewed the required number of documents or met the review sampling rate, a dialog will appear, allowing the Reviewer to click the "Mark project as complete" button.
Incomplete Review Warning
If the Reviewer has not met the review sampling rate, a dialog will appear indicating the remaining documents that need to be reviewed. The Reviewer cannot mark the project as complete until all required documents have been reviewed.
By following these guidelines, you can ensure that the review process is thoroughly completed before a project is marked as complete.
Last updated
