> For the complete documentation index, see [llms.txt](https://docs.datasaur.ai/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.datasaur.ai/advanced/extensions/review.md).

# Review

Review and resolve labeling conflicts while tracking review progress and quality.

The **Review** extension helps streamline the review process by making it easier to identify conflicts, track progress, and maintain labeling quality and consistency.

## Enable the extension

Click the gear icon in the extension panel on the right to open the **Manage extensions** dialog. Then, enable the **Review** extension.

<figure><img src="https://448889121-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-MbjY0HseEqu7LtYAt4d%2Fuploads%2Fgit-blob-f97358e8baebfc19605c3d41904e9d1c71c3547e%2FExtension%20-%20Manage%20extensions%20-%20Review.png?alt=media" alt=""><figcaption></figcaption></figure>

## How it works?

The extension helps you review labeling work by showing:

* Labeler information.
* Conflicting labels or answers.
* Review progress.

### Conflicts

Find and resolve disagreements between labelers.

* The **Document** field allows you to view conflicts in a specific document.
* The **Show work from** field allows you view conflicts made by a specific labeler.

  <figure><img src="https://448889121-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-MbjY0HseEqu7LtYAt4d%2Fuploads%2Fgit-blob-5c6f6fe8770cfb5ad8cd69048a18acdff28c62ce%2FExtension%20-%20Review%20-%20span%20labeling%20-%20highlight%20-%20show%20work%20from.png?alt=media" alt=""><figcaption></figcaption></figure>
* **Conflicts** section allows you to identify all conflicting answers provided by different labelers. The displayed information varies depending on the project type.

### Progress

This tab is only visible when the **Set review sampling rate** setting is enabled. Reviewers can use it to track their review progress and see whether they have met the configured sampling rate.

Read more information about review sampling [here](/data-studio-projects/reviewing-nlp-projects/review-sampling.md).

<figure><img src="https://448889121-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-MbjY0HseEqu7LtYAt4d%2Fuploads%2Fgit-blob-f026c769d1ab4da884fb24b3394e2e853f4884c6%2FExtension%20-%20Review%20-%20span%20labeling%20-%20sampling%20rate.png?alt=media" alt=""><figcaption></figcaption></figure>

## Span labeling

<figure><img src="https://448889121-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-MbjY0HseEqu7LtYAt4d%2Fuploads%2Fgit-blob-4697c47580f1a15164a531c1a8fe161dfcd340a7%2FExtension%20-%20Review%20-%20span%20labeling%20-%20conflicted.png?alt=media" alt=""><figcaption></figcaption></figure>

{% hint style="info" %}
For the full review workflow, see [Reviewer mode](/data-studio-projects/reviewing-nlp-projects.md).
{% endhint %}

### Conflicts

In span labeling, there are three types of conflicts. Click **View all** in each section to see the full list of conflicts.

<figure><img src="https://448889121-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-MbjY0HseEqu7LtYAt4d%2Fuploads%2Fgit-blob-065f37cde536f3c31f3235f3addd61177bd3f71c%2FExtension%20-%20Review%20-%20span%20labeling%20-%20highlight%20view%20all.png?alt=media" alt=""><figcaption></figcaption></figure>

* **Contents:** Shows text conflicts caused by edits made by labelers. For example:
  * Original text: `It is Shakespeare's longest...`
  * Edited text: `It is his longest...`

    <figure><img src="https://448889121-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-MbjY0HseEqu7LtYAt4d%2Fuploads%2Fgit-blob-650aff0daa1634fd7202e9dd3112434186c7101d%2FExtension%20-%20Review%20-%20span%20labeling%20-%20conflicted%20-%20contents.png?alt=media" alt=""><figcaption></figcaption></figure>
* **Spans:** Shows conflicting span labels created by different labelers.

  <figure><img src="https://448889121-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-MbjY0HseEqu7LtYAt4d%2Fuploads%2Fgit-blob-285ce3b3470374a14d19e7595782d0861dd7c43b%2FExtension%20-%20Review%20-%20span%20labeling%20-%20conflicted%20-%20spans.png?alt=media" alt=""><figcaption></figcaption></figure>
* **Arrows:** Shows conflicting arrow relationships created by different labelers.

  <figure><img src="https://448889121-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-MbjY0HseEqu7LtYAt4d%2Fuploads%2Fgit-blob-bd940a20ebc592fe10355c5888490b896878eade%2FExtension%20-%20Review%20-%20span%20labeling%20-%20conflicted%20-%20arrows.png?alt=media" alt=""><figcaption></figcaption></figure>

### Accept or reject all conflicted labels

Reviewers can resolve multiple remaining conflicts at once instead of reviewing each label individually. This action is available for both span and arrow conflicts.

#### Quick guide

1. Open the **Review** extension and click **View all** in **Span labels** or **Arrow labels** section.

   <figure><img src="https://448889121-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-MbjY0HseEqu7LtYAt4d%2Fuploads%2Fgit-blob-065f37cde536f3c31f3235f3addd61177bd3f71c%2FExtension%20-%20Review%20-%20span%20labeling%20-%20highlight%20view%20all.png?alt=media" alt=""><figcaption></figcaption></figure>
2. Click **Accept all** or **Reject all**.

   <figure><img src="https://448889121-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-MbjY0HseEqu7LtYAt4d%2Fuploads%2Fgit-blob-d76aab970c2bce6b953bfed1a34a020efa37ffb4%2FExtension%20-%20Review%20-%20span%20labeling%20-%20reject%20all%20accept%20all.png?alt=media" alt=""><figcaption></figcaption></figure>

{% hint style="info" %}
**Important notes**

* These actions cannot be undone.
* Only currently visible labels are affected.
* Labels connected to a sentence conflict will not be affected until the sentence conflict is resolved first.
  {% endhint %}

#### Accept all conflicted labels

* If the **Spans should have at most one label** setting is **unchecked**, all conflicted labels will be accepted.
* If the **Spans should have at most one label** setting is **checked**:
  * Labels remain conflicted if another different label exists on the same span or arrow. This requires manual review.
  * Labels are accepted only if no other different labels exist on the same span or arrow.

<figure><img src="https://448889121-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-MbjY0HseEqu7LtYAt4d%2Fuploads%2Fgit-blob-eb7be90d3ec1ba50878f3a48892dff4842d9dbab%2FProject%20settings%20-%20task%20settings%20-%20spans%20should%20have%20at%20most%20one%20label.png?alt=media" alt=""><figcaption></figcaption></figure>

#### Reject all conflicted labels

All conflicted labels will be rejected.

## Row labeling

<figure><img src="https://448889121-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-MbjY0HseEqu7LtYAt4d%2Fuploads%2Fgit-blob-ad1f7c4f87379579e358b20809f57b70468577d9%2FExtension%20-%20Review%20-%20row%20labeling%20-%20conflicted.png?alt=media" alt=""><figcaption></figcaption></figure>

{% hint style="info" %}
For the full review workflow, see [Reviewer Mode](/data-studio-projects/reviewing-nlp-projects.md).
{% endhint %}

### Conflicts

In row labeling, there is only 1 type of conflicts.

<figure><img src="https://448889121-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-MbjY0HseEqu7LtYAt4d%2Fuploads%2Fgit-blob-1e71b3424c7e19710e63e948be79f9d8a5346ba2%2FExtension%20-%20Review%20-%20row%20labeling%20-%20highlight.png?alt=media" alt=""><figcaption></figcaption></figure>

Click **View all** in the **Conflicted answers** section to see the full list of conflicts.

<figure><img src="https://448889121-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-MbjY0HseEqu7LtYAt4d%2Fuploads%2Fgit-blob-9b738282c8dec293d870dafc09f7fc64caa097ce%2FExtension%20-%20Review%20-%20row%20labeling%20-%20highlight%20-%20conflicted%20answers.png?alt=media" alt=""><figcaption></figcaption></figure>

You can also filter conflicts by labeler by selecting a labeler from the **Show work from** dropdown. This displays only the conflicts associated with that labeler.

<figure><img src="https://448889121-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-MbjY0HseEqu7LtYAt4d%2Fuploads%2Fgit-blob-ed80cde8d94c49284bd5d8fd705028fc8b4b019a%2FExtension%20-%20Review%20-%20row%20labeling%20-%20highlight%20-%20show%20work%20from.png?alt=media" alt=""><figcaption></figcaption></figure>

## Document labeling

<figure><img src="https://448889121-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-MbjY0HseEqu7LtYAt4d%2Fuploads%2Fgit-blob-717435224be4122c879333d6af8075f232a3dfd8%2FExtension%20-%20Review%20-%20document%20labeling.png?alt=media" alt=""><figcaption></figcaption></figure>

{% hint style="info" %}
For the full review workflow, see [Reviewer Mode](/data-studio-projects/reviewing-nlp-projects.md).
{% endhint %}

### Conflicts

To resolve conflicts in document labeling projects, use the **Document labeling** extension.

<figure><img src="https://448889121-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-MbjY0HseEqu7LtYAt4d%2Fuploads%2Fgit-blob-efe77bc2fdfb1fc75e58c703daa015e2ce78dc60%2FExtension%20-%20Review%20-%20document%20labeling%20-%20highlight.png?alt=media" alt=""><figcaption></figcaption></figure>

You can view answers from all labelers and resolve conflicts by selecting and submitting the best answer.

<figure><img src="https://448889121-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-MbjY0HseEqu7LtYAt4d%2Fuploads%2Fgit-blob-e0d4feab26715da1107167cad4203c15ef5f3bd4%2FExtension%20%20-%20document%20labeling%20-%20highlight%20-%20conflicted%20-%20initial.png?alt=media" alt=""><figcaption></figcaption></figure>