# Reviewing Projects

## Overview

The “Reviewer Mode” is designed to facilitate efficient and effective oversight of the labeling process. As a reviewer, your role involves ensuring the accuracy and consistency of labeled data while maintaining a smooth workflow for labelers. This mode provides you with the tools and insights you need to uphold the quality standards of your project.

## How it works?

You must have the **Reviewer** role first to use the Reviewer Mode. Roles in Datasaur can be viewed at the following [link](https://datasaurai.gitbook.io/datasaur/getting-started/lets-get-labeling#role).

### Span Labeling Reviewer Mode

You can see how conflicts in span labeling look. We have three types of conflicts in span labeling:

1. Contents conflict
2. Spans conflict
3. Arrows conflict

{% hint style="info" %}
For more information about the difference between the three types of conflict, please refer to this [link](https://datasaurai.gitbook.io/datasaur/advanced/extensions/review#conflicts-1).
{% endhint %}

You can also hover over the conflicting label between two labelers and choose the best label answer by clicking the label.

<figure><img src="https://448889121-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-MbjY0HseEqu7LtYAt4d%2Fuploads%2Fgit-blob-1f09b174620bc46010144c363856a9d3d7adb4ba%2FSpan%20Labeling%20Reviewer%20Mode.gif?alt=media" alt=""><figcaption><p>Review Project</p></figcaption></figure>

You can also Go to the next conflict and the previous conflict in the Go toolbar. Or by clicking `Alt+Shift+Right` for going to the next conflict and `Alt+Shift+Left` for going to the previous conflict.

<figure><img src="https://448889121-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-MbjY0HseEqu7LtYAt4d%2Fuploads%2Fgit-blob-d70c0078619f9420886b57ce539529aa56cb90ac%2FProject%20-%20span%20labeling%20-%20go%20menu%20-%20highlight%20go%20to%20conflicts.png?alt=media" alt=""><figcaption></figcaption></figure>

We also differentiate the label color based on the label's status

<figure><img src="https://448889121-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-MbjY0HseEqu7LtYAt4d%2Fuploads%2Fgit-blob-f662bdbb385373691ad9af0b7a3eeaee5f133816%2FProject%20-%20span%20labeling%20-%20status%20bar%20-%20reviewer%20mode%20-%20highlight%20label%20colors.png?alt=media" alt=""><figcaption></figcaption></figure>

* Your label color will be **gray** when your label has already reached **consensus** among the labelers.

<figure><img src="https://448889121-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-MbjY0HseEqu7LtYAt4d%2Fuploads%2Fgit-blob-c41b5b455f20988ce8afebc4f832029c4148c019%2FProject%20-%20span%20labeling%20-%20reviewer%20mode%20-%20hover%20on%20accepted%20by%20consensus%20label.png?alt=media" alt=""><figcaption></figcaption></figure>

* Your label color will be **yellow** when you use our [Assisted Labeling Functionality](https://datasaurai.gitbook.io/datasaur/assisted-labeling/ml-assisted-labeling); it will show you that this label comes from the Assisted Labeling Extension by making the color yellow.

<figure><img src="https://448889121-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-MbjY0HseEqu7LtYAt4d%2Fuploads%2Fgit-blob-2cfbfc178f01d0425a64291d2ef757a7aa0349a2%2FProject%20-%20span%20labeling%20-%20reviewer%20mode%20-%20hover%20on%20suggested%20label.png?alt=media" alt=""><figcaption></figcaption></figure>

* Your label color will be **blue** when your labeler and reviewer have different answers; it will show you that this label has an **incorrect** or **rejected** status by the reviewer.

<figure><img src="https://448889121-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-MbjY0HseEqu7LtYAt4d%2Fuploads%2Fgit-blob-2d04c25437bd1e957f38a7b6a276281296e1ba5c%2FProject%20-%20span%20labeling%20-%20reviewer%20mode%20-%20hover%20on%20rejected%20label.png?alt=media" alt=""><figcaption></figcaption></figure>

* Your label color will be **purple** when your reviewer labels the span; it will show you that the label is labeled by the **reviewer**.

<figure><img src="https://448889121-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-MbjY0HseEqu7LtYAt4d%2Fuploads%2Fgit-blob-5669d883e4f6107a861cadc65559527bd8e7fbfc%2FProject%20-%20span%20labeling%20-%20reviewer%20mode%20-%20hover%20on%20accepted%20label.png?alt=media" alt=""><figcaption></figcaption></figure>

* Your label color will be **red** when the status is **unresolved** or **conflicted** with another labeler; it will show you that the label is unresolved or conflicted.

<figure><img src="https://448889121-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-MbjY0HseEqu7LtYAt4d%2Fuploads%2Fgit-blob-29b36c2f1b2aa4cf51f8ada4d89e84725171ccf2%2FProject%20-%20span%20labeling%20-%20reviewer%20mode%20-%20hover%20on%20conflicted%20label.png?alt=media" alt=""><figcaption></figcaption></figure>

In span labeling, you can also see the number of span labels applied, the last labeled row, and the total solved rows. You can see it in the lower-right corner of the table display

<figure><img src="https://448889121-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-MbjY0HseEqu7LtYAt4d%2Fuploads%2Fgit-blob-475046de16b029e033deeec3a3110c6c4af46dd7%2FProject%20-%20span%20labeling%20-%20status%20bar%20-%20reviewer%20mode%20-%20highlight%20labeling%20metrics.png?alt=media" alt=""><figcaption></figcaption></figure>

### Row Labeling Reviewer Mode

<figure><img src="https://448889121-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-MbjY0HseEqu7LtYAt4d%2Fuploads%2Fgit-blob-5fac49b8e5b0951d07153d999bb010f56e5d5ba9%2FProject%20-%20row%20labeling%20-%20reviewer%20mode%20-%20row%20labeling%20extension.png?alt=media" alt=""><figcaption></figcaption></figure>

Unlike span labeling, the reviewing process in row labeling involves accepting answers within the Document and Row Labeling extension. When reviewing a row labeling project, there are two primary things:

* **Line color**

  <figure><img src="https://448889121-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-MbjY0HseEqu7LtYAt4d%2Fuploads%2Fgit-blob-e1e4811ae2397b5b77c59e3d56e3ed7b6bc3bac8%2FProject%20-%20row%20labeling%20-%20reviewer%20mode%20-%20line%20colors.png?alt=media" alt=""><figcaption></figcaption></figure>

  * **White color**: Rows containing a consensus or those already resolved by the reviewer.
  * **Pink color**: Rows containing both a consensus answer and a conflicting answer. This occurs when a question has multiple values enabled.<br>

    <figure><img src="https://448889121-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-MbjY0HseEqu7LtYAt4d%2Fuploads%2Fgit-blob-4a19341828e1a0925986d1ee9b5ee0401cc164ba%2FExtension%20-%20row%20labeling%20-%20multiple%20answers%20-%20partially%20conflicted.png?alt=media" alt=""><figcaption></figcaption></figure>
  * **Red color**: Rows without any consensus
  * **Blue color**: Selected rows.
* **Answer in the table**

  Submitting answers in the Document and Row labeling extension **will trigger the display of answers in the table**.

  * Empty answers in the table
    * No consensus
  * Answers displayed in the table
    * Meet consensus
    * Mix consensus and conflict
      * Only display the consensus answers. The conflicted answer will be displayed **after** the reviewer resolves it.
    * Resolved answers by a reviewer

### Document Labeling Reviewer Mode

<figure><img src="https://448889121-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-MbjY0HseEqu7LtYAt4d%2Fuploads%2Fgit-blob-bfcc77814f37dc9623cc702927ebebac8c29b18d%2FProject%20-%20document%20labeling%20-%20reviewer%20mode%20-%20review%20extension.png?alt=media" alt=""><figcaption></figcaption></figure>

In essence, the behavior is similar to row labeling, but let's delve into the specifics of the **Document labeling extension**. ✨

### Bounding Box Labeling Reviewer Mode

<figure><img src="https://448889121-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-MbjY0HseEqu7LtYAt4d%2Fuploads%2Fgit-blob-b474282bacb9c97ae9824584ef5d210b22a6e2d8%2FProject%20-%20bounding%20box%20labeling%20-%20reviewer%20mode%20-%20review%20extension.png?alt=media" alt=""><figcaption></figcaption></figure>

We are planning to enable the review feature for bounding box labeling in the near future. It’s coming soon!
