Review

Overview

The Review extension is designed to streamline your labeling review process, providing you with a dedicated tool to ensure the accuracy and consistency of your labeled data. Whether you're working on a small dataset or a large-scale project, this feature ensures that no labeling discrepancies go unnoticed.

How to enable it?

You just have to enable it by clicking the Manage extensions button (gear icon) from the extension panel on the right.

How it works?

This extension provides essential information related to labeling work and review, including labelers' information, conflicting labels, and reviewing progress.

This extension consists of three main tabs:

Conflicts
Member
Progress

Conflicts

There are three key aspects in this tab:

Document allows you to view the list of uploaded documents for the project and the conflict information.
Show work from allows you to select a labeler to see their specific work.
Conflicts allows you to identify all conflicting answers provided by different labelers.
1. For projects with multiple files, make sure to specify the file by selecting it from the Document dropdown.
2. The displayed information varies depending on the project type.

Progress

This tab will only be visible when the Set review sampling rate setting is enabled. Reviewers can track their progress in this tab, determining whether they are still in progress or have fulfilled the configured sampling rate.

Read more information about review sampling here.

Span Based Labeling

For the detailed reviewing process, please refer to this Reviewer mode page.

Conflicts

The labeling conflicts in span labeling will be shown under the show work from dropdown.

It will be separated into three sections:

Contents: This will show you the conflicted words that were changed by the labeler, for example:
1. The original sentence is “It is Shakespeare's longest...”
2. A labeler edits the sentence into “It is his longest...”
Spans: This will show you the conflicted spans that were labeled by the labelers.
Arrows: This will show you the conflicted arrows that were labeled by the labelers.

Accept or Reject All Conflicted Labels

Previously, managing a large number of unresolved or conflicting labels was time-consuming for Reviewers, as they had to manually accept or reject each label individually. This capability streamlines the process, allowing Reviewers to accept or reject all remaining labels with a single action.

How to: Accept or Reject All

This action can be applied to both Span and Arrow conflicts.

Navigate to the Review extension and then click the View all button.

Click Accept all or Reject all depending on the action you would like to perform.

Important Notes

Both actions cannot be undone and only apply to labels that are currently visible in the reviewers' interface.
Accepting or rejecting labels will not affect labels associated with a span or arrow that is part of a sentence in conflict. The correct sentence must be selected first by the reviewer.

Accept All Conflicted Labels

When the Spans should have at most one label setting is unchecked, all conflicted labels will be accepted.
When the Spans should have at most one label setting is checked, then:
- If a conflicted label belongs to a span/arrow that has another different label applied to it, then the label will remain conflicted. These require manual resolution by the reviewer.
- If a conflicted label belongs to a span/arrow that has no other different label applied to it, then the label will be accepted.

Reject All Conflicted Labels

All conflicted labels will be rejected.

Row Based Labeling

For the detailed reviewing process, please refer to this Reviewer Mode page

Conflicts

The labeling conflicts in row based labeling will be shown under the show work from dropdown.

You can click the View all button to see the conflicted answers that occur in the labeling process. It will show you the list of conflicting labels.

You can also filter the conflicting answers based on the labeler. To do that, you have to select which labeler you want to review in the Show work from dropdown.