Review
Last updated
Last updated
The “Review Extension” is designed to streamline your labeling review process, providing you with a dedicated tool to ensure the accuracy and consistency of your labeled data. Whether you're working on a small dataset or a large-scale project, this feature ensures that no labeling discrepancies go unnoticed.
You just have to enable it by clicking the Manage Extension button in the right sidebar.
This extension provides essential information related to labeling work and review, including labelers' information, conflicting labels, and reviewing progress.
This extension consists of three main tabs:
Conflicts
Member
Progress
There are three key aspects in this tab:
Document allows you to view the list of uploaded documents for the project and the conflict information.
Show work from allows to select a labeler to see their specific work.\
Conflicts allows you to identify all conflicting answers provided by different labelers.
For projects with multiple files, make sure to specify the file by selecting it from the Document dropdown.
The displayed information varies depending on the project type.
You can easily access the list of labelers assigned to the project and their labeling status. Each project type has different displayed information.
This tab will only be visible when the “Set review sampling rate” setting is enabled. Reviewers can track their progress in this tab, determining whether they are still in progress or have fulfilled the configured sampling rate.
Read more information about Review Sampling here.
For the detailed how reviewing process, please refer to this Reviewer Mode page.
The Labeling Conflicts in Span Based labeling will be shown under the show work from dropdown.
It will be separated into three sections:
Contents: This will show you the conflicted words that were changed by the labeler, for example:
The original sentence is “the quick brown fox.”
Labeler 1: edit the sentence into “the quick black fox.”
Labeler 2: edit the sentence into “the quick fox.”
Spans: This will show you the conflicted spans that were labeled by the labelers.
Arrows: This will show you the conflicted arrows that were labeled by the labelers.
Previously, managing a large number of unresolved or conflicting labels was time-consuming for Reviewers, as they had to manually accept or reject each label individually. This capability streamlines the process, allowing Reviewers to accept or reject all remaining labels with a single action.
This action can be applied to both Span and Arrow conflicts.
Navigate to the Review extension and then click the "View All" button.
Click "Accept All" or "Reject All" depending on the action you would like to perform.
Important Notes
Both actions cannot be undone and only apply to labels that are currently visible in the Reviewer’s interface.
Accepting or rejecting labels will not affect labels associated with a span or arrow that is part of a sentence in conflict. The correct sentence must be selected first by the Reviewer.
When the project setting “Spans should have at most one label” is unchecked, all conflicted labels will be accepted.
When the project setting “Spans should have at most one label” is checked, then:
If a conflicted label belongs to a span/arrow that has another different label applied to it, then the label will remain conflicted. These require manual resolution by the Reviewer.
If a conflicted label belongs to a span/arrow that has no other different label applied to it, then the label will be accepted.
All conflicted labels will be rejected.
In the members tab, you can see your labeler’s progress. It will also show how many files they have already labeled.
For the detailed how reviewing process, please refer to this Reviewer Mode page
The Labeling Conflicts in Row Based labeling will be shown under the show work from dropdown.
You can click the View All button to see the Conflicted Answers that occur in the labeling process. And it will show you the list of conflicting labels.
You can also filter the conflicting answers based on the labeler. To do that, you have to select which labeler you want to review in the Show Work From the dropdown.
This will show you the conflicting answer caused by the labeler.
In the members tab, you can see your labeler’s progress.
For the detailed how reviewing process, please refer to this Reviewer Mode page
To resolve conflicts in Doc Based Labeling you have to go to Document Labeling extension to resolve conflicted answers.
You can see all the labeler’s answers. And you can also resolve the conflicts by choosing the best answer and submitting it.
In the members tab, you can see your labeler’s progress. It will also show how many files they have already labeled.
