Reviewing Projects

Overview

Reviewer mode helps you review labeling work and ensure data quality. As a reviewer, you check accuracy, resolve conflicts, and keep the workflow consistent.

How it works?

You must have the Reviewer role to use the reviewer mode in a project. Learn more about roles here.

Reviewer mode for span labeling

In span labeling projects, there are three types of conflicts:

• Content conflict

• Span conflict

• Arrow conflict

See the documentation for details on the differences between these conflict types.

Resolve conflicts

To resolve a conflict, right-click the conflicted label to accept or reject it. You can also replace it by left-clicking the label and selecting a new one.

Navigate between conflicts

You can navigate between conflicts from the Go menu or by pressing Alt + Shift + Right to go to the next conflict and Alt + Shift + Left to go to the previous one.

Label status colors

In reviewer mode, label colors indicate the status of each label. You can view the color legend in the status bar at the bottom.

• Gray labels mean they have reached consensus among labelers.

  
• Yellow labels indicate that they were created using the asisted labeling extensions.

  
• Blue labels mean the label has been rejected by the reviewer. By default, rejected labels are hidden. You can make them visible in File → Settings → Administration by enabling Show rejected labels in reviewer mode.

  
• Purple labels indicate that they were applied by the reviewer.

  
• Red labels indicate that they are conflicted (have not reached the consensus threshold) and require review to be accepted or rejected.

  

In span labeling, you can see the number of applied labels, the last labeled row, and the total resolved rows. This information is shown in the bottom-right corner.

Reviewer mode for row labeling

Reviewing in row labeling focuses on accepting, rejecting, or updating answers in each row using the Row labeling extension.

Row colors

Each row has a colored line on the left side of the number column to indicate its status.

• No colored line means the answers for that row have reached consensus among labelers or have already been resolved by the reviewer.

• A red line indicates that the row has no consensus. All answers are conflicted.

• A pink line indicates that the row contains both a consensus answer and a conflicting answer. This occurs when a question allows multiple answers.

  

Answers in the table

Whether answers are shown in the table depends on their status.

• Meets consensus: Answers are displayed.

• Conflicted (No consensus): No answers are displayed.

• Mixed consensus and conflict: Only consensus answers are shown; conflicted answers appear after review.

• Resolved by reviewer: Final answers are displayed after review.

Submitting answers in the Row labeling extension updates the table accordingly.

Reviewer mode for document labeling

The behavior is similar to row labeling, with the main interactions happening in the Document labeling extension.

Reviewer mode for bounding box labeling

Review for bounding box labeling is not yet available. This feature is coming soon!