> For the complete documentation index, see [llms.txt](https://docs.datasaur.ai/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.datasaur.ai/advanced/extensions/review.md).

# Review

The **Review** extension helps streamline the review process by making it easier to identify conflicts, track progress, and maintain labeling quality and consistency.

## Enable the extension

Click the gear icon in the extension panel on the right to open the **Manage extensions** dialog. Then, enable the **Review** extension.

<figure><img src="/files/KdE4sr2zT1NGxWS5vXNd" alt=""><figcaption></figcaption></figure>

## How it works?

The extension helps you review labeling work by showing:

* Labeler information.
* Conflicting labels or answers.
* Review progress.

### Conflicts

Find and resolve disagreements between labelers.

* The **Document** field allows you to view conflicts in a specific document.
* The **Show work from** field allows you view conflicts made by a specific labeler.

  <figure><img src="/files/KbZ8LdHjhjtm8o0SEoDW" alt=""><figcaption></figcaption></figure>
* **Conflicts** section allows you to identify all conflicting answers provided by different labelers. The displayed information varies depending on the project type.

### Progress

This tab is only visible when the **Set review sampling rate** setting is enabled. Reviewers can use it to track their review progress and see whether they have met the configured sampling rate.

Read more information about review sampling [here](/data-studio-projects/reviewing-nlp-projects/review-sampling.md).

<figure><img src="/files/V9twNr8XVYogoEZrsSA0" alt=""><figcaption></figcaption></figure>

## Span labeling

<figure><img src="/files/wjZ2DNBHiroGLcTUIK4o" alt=""><figcaption></figcaption></figure>

{% hint style="info" %}
For the full review workflow, see [Reviewer mode](/data-studio-projects/reviewing-nlp-projects.md).
{% endhint %}

### Conflicts

In span labeling, there are three types of conflicts. Click **View all** in each section to see the full list of conflicts.

<figure><img src="/files/iyvi4fZOy4LM9oRqx746" alt=""><figcaption></figcaption></figure>

* **Contents:** Shows text conflicts caused by edits made by labelers. For example:
  * Original text: `It is Shakespeare's longest...`
  * Edited text: `It is his longest...`

    <figure><img src="/files/XxO1K1CQ3QvtxUTZs6Ns" alt=""><figcaption></figcaption></figure>
* **Spans:** Shows conflicting span labels created by different labelers.

  <figure><img src="/files/bOR1qbDxl7tYvi309te4" alt=""><figcaption></figcaption></figure>
* **Arrows:** Shows conflicting arrow relationships created by different labelers.

  <figure><img src="/files/tumXMjCMkci5JSHp2uXx" alt=""><figcaption></figcaption></figure>

### Accept or reject all conflicted labels

Reviewers can resolve multiple remaining conflicts at once instead of reviewing each label individually. This action is available for both span and arrow conflicts.

#### Quick guide

1. Open the **Review** extension and click **View all** in **Span labels** or **Arrow labels** section.

   <figure><img src="/files/iyvi4fZOy4LM9oRqx746" alt=""><figcaption></figcaption></figure>
2. Click **Accept all** or **Reject all**.

   <figure><img src="/files/5ZmFaJjdQKrUq1KQz2qw" alt=""><figcaption></figcaption></figure>

{% hint style="info" %}
**Important notes**

* These actions cannot be undone.
* Only currently visible labels are affected.
* Labels connected to a sentence conflict will not be affected until the sentence conflict is resolved first.
  {% endhint %}

#### Accept all conflicted labels

* If the **Spans should have at most one label** setting is **unchecked**, all conflicted labels will be accepted.
* If the **Spans should have at most one label** setting is **checked**:
  * Labels remain conflicted if another different label exists on the same span or arrow. This requires manual review.
  * Labels are accepted only if no other different labels exist on the same span or arrow.

<figure><img src="/files/SQ65YnkK5FDQQAkTQ952" alt=""><figcaption></figcaption></figure>

#### Reject all conflicted labels

All conflicted labels will be rejected.

## Row labeling

<figure><img src="/files/rUKSGKJuD3nzHBxgKEzW" alt=""><figcaption></figcaption></figure>

{% hint style="info" %}
For the full review workflow, see [Reviewer Mode](/data-studio-projects/reviewing-nlp-projects.md).
{% endhint %}

### Conflicts

In row labeling, there is only 1 type of conflicts.

<figure><img src="/files/A7eTj4oGj8DFqqJnCovF" alt=""><figcaption></figcaption></figure>

Click **View all** in the **Conflicted answers** section to see the full list of conflicts.

<figure><img src="/files/UbFLNs6ictvszUtt8OZt" alt=""><figcaption></figcaption></figure>

You can also filter conflicts by labeler by selecting a labeler from the **Show work from** dropdown. This displays only the conflicts associated with that labeler.

<figure><img src="/files/h0fFq96eFTonVxlleQz9" alt=""><figcaption></figcaption></figure>

## Document labeling

<figure><img src="/files/0dZu3Tt31f0LG4bKyY3C" alt=""><figcaption></figcaption></figure>

{% hint style="info" %}
For the full review workflow, see [Reviewer Mode](/data-studio-projects/reviewing-nlp-projects.md).
{% endhint %}

### Conflicts

To resolve conflicts in document labeling projects, use the **Document labeling** extension.

<figure><img src="/files/u587QNznIeboW62yTpRC" alt=""><figcaption></figcaption></figure>

You can view answers from all labelers and resolve conflicts by selecting and submitting the best answer.

<figure><img src="/files/icsDqJfDdNqSQQCRFxhM" alt=""><figcaption></figcaption></figure>