Review

Review and resolve labeling conflicts while tracking review progress and quality.

The Review extension helps streamline the review process by making it easier to identify conflicts, track progress, and maintain labeling quality and consistency.

Enable the extension

Click the gear icon in the extension panel on the right to open the Manage extensions dialog. Then, enable the Review extension.

How it works?

The extension helps you review labeling work by showing:

Labeler information.
Conflicting labels or answers.
Review progress.

Conflicts

Find and resolve disagreements between labelers.

The Document field allows you to view conflicts in a specific document.
The Show work from field allows you view conflicts made by a specific labeler.
Conflicts section allows you to identify all conflicting answers provided by different labelers. The displayed information varies depending on the project type.

Progress

This tab is only visible when the Set review sampling rate setting is enabled. Reviewers can use it to track their review progress and see whether they have met the configured sampling rate.

Read more information about review sampling here.

Span labeling

For the full review workflow, see Reviewer mode.

Conflicts

In span labeling, there are three types of conflicts. Click View all in each section to see the full list of conflicts.

Contents: Shows text conflicts caused by edits made by labelers. For example:
- Original text: It is Shakespeare's longest...
- Edited text: It is his longest...
Spans: Shows conflicting span labels created by different labelers.
Arrows: Shows conflicting arrow relationships created by different labelers.

Accept or reject all conflicted labels

Reviewers can resolve multiple remaining conflicts at once instead of reviewing each label individually. This action is available for both span and arrow conflicts.

Quick guide

Open the Review extension and click View all in Span labels or Arrow labels section.
Click Accept all or Reject all.

Important notes

These actions cannot be undone.
Only currently visible labels are affected.
Labels connected to a sentence conflict will not be affected until the sentence conflict is resolved first.

Accept all conflicted labels

If the Spans should have at most one label setting is unchecked, all conflicted labels will be accepted.
If the Spans should have at most one label setting is checked:
- Labels remain conflicted if another different label exists on the same span or arrow. This requires manual review.
- Labels are accepted only if no other different labels exist on the same span or arrow.

Reject all conflicted labels

All conflicted labels will be rejected.

Row labeling

For the full review workflow, see Reviewer Mode.

Conflicts

In row labeling, there is only 1 type of conflicts.

Click View all in the Conflicted answers section to see the full list of conflicts.

You can also filter conflicts by labeler by selecting a labeler from the Show work from dropdown. This displays only the conflicts associated with that labeler.