# Review ## Overview The **Review** extension is designed to streamline your labeling review process, providing you with a dedicated tool to ensure the accuracy and consistency of your labeled data. Whether you're working on a small dataset or a large-scale project, this feature ensures that no labeling discrepancies go unnoticed. ## How to enable it? You just have to **enable** it by clicking the **Manage extensions** button (gear icon) from the extension panel on the right.

## How it works? This extension provides essential information related to labeling work and review, including **labelers' information, conflicting labels,** and **reviewing progress**. This extension consists of three main tabs: 1. Conflicts 2. Member 3. Progress ### Conflicts There are three key aspects in this tab: 1. **Document** allows you to view the list of uploaded documents for the project and the conflict information. 2. **Show work from** allows you to select a labeler to see their specific work.

3. **Conflicts** allows you to identify all conflicting answers provided by different labelers. 1. For projects with multiple files, make sure to specify the file by selecting it from the Document dropdown. 2. The displayed information varies depending on the project type. ### Progress This tab will only be visible when the **Set review sampling rate** setting is enabled. Reviewers can track their progress in this tab, determining whether they are still in progress or have fulfilled the configured sampling rate. Read more information about review sampling [here](https://docs.datasaur.ai/data-studio-projects/reviewing-nlp-projects/review-sampling).

## Span Based Labeling

{% hint style="info" %} For the detailed reviewing process, please refer to this [Reviewer mode page](https://datasaurai.gitbook.io/datasaur/getting-started/reviewer-mode#token-labeling-reviewer-mode). {% endhint %} ### Conflicts The labeling conflicts in span labeling will be shown under the show work from dropdown.

It will be separated into three sections: 1. **Contents:** This will show you the conflicted words that were changed by the labeler, for example: 1. The original sentence is “It is **Shakespeare's** longest...” 2. A labeler edits the sentence into “It is **his** longest...”

2. **Spans:** This will show you the conflicted spans that were labeled by the labelers.

3. **Arrows:** This will show you the conflicted arrows that were labeled by the labelers.

### Accept or Reject All Conflicted Labels Previously, managing a large number of unresolved or conflicting labels was time-consuming for Reviewers, as they had to manually accept or reject each label individually. This capability streamlines the process, allowing Reviewers to accept or reject all remaining labels with a single action. #### How to: Accept or Reject All {% hint style="info" %} This action can be applied to both Span and Arrow conflicts. {% endhint %} Navigate to the Review extension and then click the **View all** button.

Click **Accept all** or **Reject all** depending on the action you would like to perform.

{% hint style="info" %} **Important Notes** * Both actions **cannot be undone** and only apply to labels that are currently visible in the reviewers' interface. * Accepting or rejecting labels will not affect labels associated with a span or arrow that is part of a sentence in conflict. The correct sentence must be selected first by the reviewer. {% endhint %} #### Accept All Conflicted Labels

* When the **Spans should have at most one label** setting is **unchecked**, all conflicted labels will be accepted. * When the **Spans should have at most one label** setting is **checked**, then: * If a conflicted label belongs to a span/arrow that **has another different label applied** to it, then the label will remain conflicted. These require manual resolution by the reviewer. * If a conflicted label belongs to a span/arrow that **has no other different label applied** to it, then the label will be accepted. #### Reject All Conflicted Labels All conflicted labels will be rejected. ## Row Based Labeling

{% hint style="info" %} For the detailed reviewing process, please refer to this [Reviewer Mode page](https://datasaurai.gitbook.io/datasaur/getting-started/reviewer-mode#row-labeling-reviewer-mode) {% endhint %} ### Conflicts The labeling conflicts in row based labeling will be shown under the show work from dropdown.

You can click the **View all** button to see the conflicted answers that occur in the labeling process. It will show you the list of conflicting labels.

You can also filter the conflicting answers based on the labeler. To do that, you have to select which labeler you want to review in the **Show work** **from** dropdown.

This will show you the conflicting answers caused by the labeler. ## Document Labeling

{% hint style="info" %} For the detailed reviewing process, please refer to this [Reviewer mode page](https://datasaurai.gitbook.io/datasaur/getting-started/reviewer-mode#document-labeling-reviewer-mode). {% endhint %} ### Conflicts To resolve conflicts in document labeling projects, you have to go to the **Document labeling** extension to resolve conflicted answers.

You can see all labelers' answers and you can also resolve the conflicts by choosing the best answer and submitting it.