Datasaur
Visit our websitePricingBlogPlaygroundAPI Docs
  • Welcome to Datasaur
    • Getting started with Datasaur
  • Data Studio Projects
    • Labeling Task Types
      • Span Based
        • OCR Labeling
        • Audio Project
      • Row Based
      • Document Based
      • Bounding Box
      • Conversational
      • Mixed Labeling
      • Project Templates
        • Test Project
    • Creating a Project
      • Data Formats
      • Data Samples
      • Split Files
      • Consensus
      • Dynamic Review Capabilities
    • Pre-Labeled Project
    • Let's Get Labeling!
      • Span Based
        • Span + Line Labeling
      • Row & Document Based
      • Bounding Box Labeling
      • Conversational Labeling
      • Label Sets / Question Sets
        • Dynamic Question Set
      • Multiple Label Sets
    • Reviewing Projects
      • Review Sampling
    • Adding Documents to an Ongoing Project
    • Export Project
  • LLM Projects
    • LLM Labs Introduction
    • Sandbox
      • Direct Access LLMs
      • File Attachment
      • Conversational Prompt
    • Deployment
      • Deployment API
    • Knowledge base
      • External Object Storage
      • File Properties
    • Models
      • Amazon SageMaker JumpStart
      • Amazon Bedrock
      • Open AI
      • Azure OpenAI
      • Vertex AI
      • Custom model
      • Fine-tuning
      • LLM Comparison Table
    • Evaluation
      • Automated Evaluation
        • Multi-application evaluation
        • Custom metrics
      • Ranking (RLHF)
      • Rating
      • Performance Monitoring
    • Dataset
    • Pricing Plan
  • Workspace Management
    • Workspace
    • Role & Permission
    • Analytics
      • Inter-Annotator Agreement (IAA)
        • Cohen's Kappa Calculation
        • Krippendorff's Alpha Calculation
      • Custom Report Builder
      • Project Report
      • Evaluation Metrics
    • Activity
    • File Transformer
      • Import Transformer
      • Export Transformer
      • Upload File Transformer
      • Running File Transformer
    • Label Management
      • Label Set Management
      • Question Set Management
    • Project Management
      • Self-Assignment
        • Self-Unassign
      • Transfer Assignment Ownership
      • Reset Labeling Work
      • Mark Document as Complete
      • Project Status Workflow
        • Read-only Mode
      • Comment Feature
      • Archive Project
    • Automation
      • Action: Create Projects
  • Assisted Labeling
    • ML Assisted Labeling
      • Amazon Comprehend
      • Amazon SageMaker
      • Azure ML
      • CoreNLP NER
      • CoreNLP POS
      • Custom API
      • FewNERD
      • Google Vertex AI
      • Hugging Face
      • LLM Assisted Labeling
        • Prompt Examples
        • Custom Provider
      • LLM Labs (beta)
      • NLTK
      • Sentiment Analysis
      • spaCy
      • SparkNLP NER
      • SparkNLP POS
    • Data Programming
      • Example of Labeling Functions
      • Labeling Function Analysis
      • Inter-Annotator Agreement for Data Programming
    • Predictive Labeling
  • Assisted Review
    • Label Error Detection
  • Building Your Own Model
    • Datasaur Dinamic
      • Datasaur Dinamic with Hugging Face
      • Datasaur Dinamic with Amazon SageMaker Autopilot
  • Advanced
    • Script-Generated Question
    • Shortcuts
    • Extensions
      • Labels
      • Review
      • Document and Row Labeling
      • Bounding Box Labels
      • List of Files
      • Comments
      • Analytics
      • Dictionary
      • Search
      • Labeling Guidelines
      • Metadata
      • Grammar Checker
      • ML Assisted Labeling
      • Data Programming
      • Datasaur Dinamic
      • Predictive Labeling
      • Label Error Detection
      • LLM Sandbox
    • Tokenizers
  • Integrations
    • External Object Storage
      • AWS S3
        • With IRSA
      • Google Cloud Storage
      • Azure Blob Storage
      • Dropbox
    • SAML
      • Okta
      • Microsoft Entra ID
    • SCIM
      • Okta
      • Microsoft Entra ID
    • Webhook Notifications
      • Webhook Signature
      • Events
      • Custom Headers
    • Robosaur
      • Commands
        • Create Projects
        • Apply Project Tags
        • Export Projects
        • Generate Time Per Task Report
        • Split Document
      • Storage Options
  • API
    • Datasaur APIs
    • Credentials
    • Create Project
      • New mutation (createProject)
      • Python Script Example
    • Adding Documents
    • Labeling
      • Create Label Set
      • Add Label Sets into Existing Project
      • Get List of Label Sets in a Project
      • Add Label Set Item into Project's Label Set
      • Programmatic API Labeling
      • Inserting Span and Arrow Label into Document
    • Export Project
      • Custom Webhook
    • Get Data
      • Get List of Projects
      • Get Document Information
      • Get List of Tags
      • Get Cabinet
      • Export Team Overview
      • Check Job
    • Custom OCR
      • Importable Format
    • Custom ASR
    • Run ML-Assisted Labeling
  • Security and Compliance
    • Security and Compliance
      • 2FA
  • Compatibility & Updates
    • Common Terminology
    • Recommended Machine Specifications
    • Supported Formats
    • Supported Languages
    • Release Notes
      • Version 6
        • 6.112.0
        • 6.111.0
        • 6.110.0
        • 6.109.0
        • 6.108.0
        • 6.107.0
        • 6.106.0
        • 6.105.0
        • 6.104.0
        • 6.103.0
        • 6.102.0
        • 6.101.0
        • 6.100.0
        • 6.99.0
        • 6.98.0
        • 6.97.0
        • 6.96.0
        • 6.95.0
        • 6.94.0
        • 6.93.0
        • 6.92.0
        • 6.91.0
        • 6.90.0
        • 6.89.0
        • 6.88.0
        • 6.87.0
        • 6.86.0
        • 6.85.0
        • 6.84.0
        • 6.83.0
        • 6.82.0
        • 6.81.0
        • 6.80.0
        • 6.79.0
        • 6.78.0
        • 6.77.0
        • 6.76.0
        • 6.75.0
        • 6.74.0
        • 6.73.0
        • 6.72.0
        • 6.71.0
        • 6.70.0
        • 6.69.0
        • 6.68.0
        • 6.67.0
        • 6.66.0
        • 6.65.0
        • 6.64.0
        • 6.63.0
        • 6.62.0
        • 6.61.0
        • 6.60.0
        • 6.59.0
        • 6.58.0
        • 6.57.0
        • 6.56.0
        • 6.55.0
        • 6.54.0
        • 6.53.0
        • 6.52.0
        • 6.51.0
        • 6.50.0
        • 6.49.0
        • 6.48.0
        • 6.47.0
        • 6.46.0
        • 6.45.0
        • 6.44.0
        • 6.43.0
        • 6.42.0
        • 6.41.0
        • 6.40.0
        • 6.39.0
        • 6.38.0
        • 6.37.0
        • 6.36.0
        • 6.35.0
        • 6.34.0
        • 6.33.0
        • 6.32.0
        • 6.31.0
        • 6.30.0
        • 6.29.0
        • 6.28.0
        • 6.27.0
        • 6.26.0
        • 6.25.0
        • 6.24.0
        • 6.23.0
        • 6.22.0
        • 6.21.0
        • 6.20.0
        • 6.19.0
        • 6.18.0
        • 6.17.0
        • 6.16.0
        • 6.15.0
        • 6.14.0
        • 6.13.0
        • 6.12.0
        • 6.11.0
        • 6.10.0
        • 6.9.0
        • 6.8.0
        • 6.7.0
        • 6.6.0
        • 6.5.0
        • 6.4.0
        • 6.3.0
        • 6.2.0
        • 6.1.0
        • 6.0.0
      • Version 5
        • 5.63.0
        • 5.62.0
        • 5.61.0
        • 5.60.0
  • Deployment
    • Self-Hosted
      • AWS Marketplace
        • Data Studio
        • LLM Labs
Powered by GitBook
On this page
  • Overview
  • How it works?
  • Span Labeling Reviewer Mode
  • Row Labeling Reviewer Mode
  • Document Labeling Reviewer Mode
  • Bounding Box Labeling Reviewer Mode
  • LLM Evaluation Reviewer Mode
  • LLM Ranking Reviewer Mode
  1. Data Studio Projects

Reviewing Projects

Last updated 10 months ago

Overview

The “Reviewer Mode” is designed to facilitate efficient and effective oversight of the labeling process. As a reviewer, your role involves ensuring the accuracy and consistency of labeled data while maintaining a smooth workflow for labelers. This mode provides you with the tools and insights you need to uphold the quality standards of your project.

How it works?

You must have the Reviewer role first to use the Reviewer Mode. Roles in Datasaur can be viewed at the following .

Span Labeling Reviewer Mode

You can see how conflicts in token labeling look. We have three types of conflicts in token labeling:

  1. Contents conflict

  2. Spans conflict

  3. Arrows conflict

For more information about the difference between the three types of conflict, please refer to this .

You can also hover over the conflicting label between two labelers and choose the best label answer by clicking the label.

You can also Go to the next conflict and the previous conflict in the Go toolbar. Or by clicking Alt+Shift+Right for going to the next conflict and Alt+Shift+Left for going to the previous conflict.

We also differentiate the label color based on the label's status

  • Your label color will be gray when your label has already reached consensus among the labelers.

  • Your label color will be blue when your labeler and reviewer have different answers; it will show you that this label has an incorrect or rejected status by the reviewer.

  • Your label color will be purple when your reviewer labels the token; it will show you that the label is labeled by the reviewer.

  • Your label color will be red when the status is unresolved or conflicted with another labeler; it will show you that the label is unresolved or conflicted.

In token labeling, you can also see the number of token labels applied, the last labeled row, and the total solved rows. You can see it in the lower-right corner of the table display

Row Labeling Reviewer Mode

Unlike token labeling, the reviewing process in row labeling involves accepting answers within the Document and Row Labeling extension. When reviewing a row labeling project, there are two primary things:

Table

  • Line color

    • White color: Rows containing a consensus or those already resolved by the reviewer.

    • Red color: Rows without any consensus

    • Blue color: Selected rows.

  • Answer in the table

    Submitting answers in the Document and Row labeling extension will trigger the display of answers in the table.

    • Empty answers in the table

      • No consensus

    • Answers displayed in the table

      • Meet consensus

      • Mix consensus and conflict

        • Only display the consensus answers. The conflicted answer will be displayed after the reviewer resolves it.

      • Resolved answers by a reviewer

Document and row labeling extensions

  • Consensus rows

    • Answers are in shown blue-colored labels

    • Answers are selected in the question field

  • Mix consensus and conflict rows

  • Conflict answers are shown in the red-colored labels

  • Consensus answers are shown in blue-colored labels and selected

Document Labeling Reviewer Mode

In essence, the behavior is similar to row labeling, but let's delve into the specifics of the Document labeling extension. ✨

Bounding Box Labeling Reviewer Mode

We are planning to enable the review feature for bounding box labeling in the near future. It’s coming soon!

LLM Evaluation Reviewer Mode

The “Reviewer Mode” is designed to facilitate efficient and effective oversight of the labeling process. As a reviewer, your role involves ensuring the accuracy and consistency of labeled data while maintaining a smooth workflow for labelers. This mode provides you with the tools and insights you need to uphold the quality standards of your project.

Reviewer can input their own rating and answer by click the Edit button and edit the answer (and rating) the click the submit button on the bottom.

Reviewer also able to select the labelers’ answer as shown on the image below. Click the submit button after you select one of the labelers’ answer.

LLM Evaluation Report

When you click “Mark Project as Complete” in the LLM Evaluation project on Reviewer mode, a window appears. This window includes the LLM Evaluation Report Card and a prompt to download the 'LLM Evaluation Format.csv'. The calculations within this report are derived from the reviewers' answers.

LLM Evaluation Report Card

The LLM Evaluation Report Card consists of:

  1. Average Rating Score (0.00 - 1.00): This score is the average of all prompts rated across all files. Datasaur rounds the LLM Evaluation Score: values equal to or greater than 0.5 are rounded up, while values less than 0.5 are rounded down.

  2. Rating Distribution Bar Chart: This chart visualizes the distribution of 1–5 star ratings and includes a section for unrated items.

  3. Unrated Definition: “Unrated” refers to prompts that have not received any reviewers' answers.

When you export all files, a new file named 'llm-evaluation-report-card.csv' will be created, containing all of the information above.

LLM Ranking Reviewer Mode

The “Reviewer Mode” is designed to facilitate efficient and effective oversight of the labeling process. As a reviewer, your role involves ensuring the accuracy and consistency of labeled data while maintaining a smooth workflow for labelers. This mode provides you with the tools and insights you need to uphold the quality standards of your project.

Reviewer can input their own ranking then click the submit button on the bottom.

Reviewer also able to select the labelers’ answer as shown on the image below. Click the submit button after you select one of the labelers’ answer.

Your label color will be yellow when you use our ; it will show you that this label comes from the Assisted Labeling Extension by making the color yellow.

Yellow color: Rows containing both a consensus answer and a conflicting answer. This occurs when a question has multiple values enabled.

Assisted Labeling Functionality
link
link