# Getting started with Datasaur Datasaur is a private AI platform for regulated industries. These docs cover Data Studio (data labeling), LLM Labs, and self-hosted deployment. Welcome to Datasaur, a web-based platform for managing data labeling workflows for Data Studio and LLM projects.\ \ The platform provides tools to support efficient and accurate labeling across multiple data types. It also includes advanced LLM tools designed to support collaboration, improve workflow efficiency, and help you build higher-quality language models securely. ### Data Studio Data labeling is a key step in training supervised learning AI models. A labeling workflow typically includes three main components: the data to label, the label set (also known as the ontology or taxonomy), and the people performing the labeling. Datasaur s a web-based platform for managing data labeling workflows, including importing data, applying labels, and collaborating with your labeling team. The platform is designed to improve labeling efficiency and quality while maintaining data security. The platform also includes AI-powered features to help automate the labeling process. Datasaur supports common Natural Language Processing (NLP) labeling workflows, including: * [Span labeling](/data-studio-projects/nlp-task-types/span-based) * [Textual / row classification](/data-studio-projects/nlp-task-types/row-based) * [Document classification](/data-studio-projects/nlp-task-types/document-based) * [OCR labeling](/data-studio-projects/nlp-task-types/span-based/ocr-labeling) * [Bounding box labeling](/data-studio-projects/nlp-task-types/bounding-box) * [Audio labeling](/data-studio-projects/nlp-task-types/span-based/audio-project) * [Conversational labeling](/data-studio-projects/nlp-task-types/conversational) {% hint style="info" %} To get started with Data Studio, see the [Data Studio documentation](/data-studio-projects/data-studio-introduction). {% endhint %} ### LLM Labs Building and improving LLMs typically involves several key components: experimenting with prompts and models, managing knowledge and context for retrieval, and evaluating outputs for quality and accuracy. In LLM Labs, you can experiment with different LLMs, manage knowledge bases for semantic search and RAG workflows, evaluate model outputs through human feedback and automated scoring, and streamline AI development workflows in one place. The platform is designed to improve efficiency, collaboration, and output quality while maintaining data security. {% hint style="info" %} To get started with LLM Labs, see the [LLM Labs documentation](/llm-projects/llm-introduction). {% endhint %} ### Workspace management Effective workspace management is an important part of successful data labeling and LLM development workflows. Datasaur provides tools for managing teams, reviewing work quality, and monitoring project activity through reports and QA workflows. {% hint style="info" %} To get started with workspace management, see the [Workspace documentation](/workspace-management/workspace). {% endhint %} \ More questions? Email # Data Studio Introduction Data Studio is Datasaur’s data-labeling platform, one part of the data pipeline behind Datasaur’s private AI deployments. Welcome to **Data Studio!** This guide introduces the essential tools and features available to help you manage and analyze your datasets effectively. Whether you're just getting started or looking to deepen your usage, this documentation will walk you through the process. All activities in Data Studio are organized within a workspace. This workspace is your command center for uploading data, labeling, transforming, reviewing, and exporting. It’s also where collaboration happens, you can invite your team, assign roles, and manage everything in one place. ## What you can do in Data Studio ### Data Ingestion Easily upload data from multiple sources, including text, CSV, and JSON files using the [Project Creation Wizard](/data-studio-projects/creating-a-project). You can also [add additional documents](/workspace-management/project-management/manage-documents-in-an-ongoing-project) later or connect to [external storage](/integrations/external-object-storage) or other integrations. ### Transformation & Labeling Choose from multiple [labeling task type](/data-studio-projects/nlp-task-types) including Span, Row, Document and more. You can label manually, import [pre-labeled data](/data-studio-projects/pre-labeled-project) or use one of the Assisted Labeling option including [ML Assisted Labeling](/assisted-labeling/ml-assisted-labeling) to speed up the process with AI suggestions. ### Collaboration Easily work with others by sharing [workspaces](/workspace-management/workspace) and [assigning roles](/workspace-management/role-and-permission). Control access and manage projects in one place. ### Automation & Insights Enable [automation](/workspace-management/automation) to streamline repetitive tasks. Built-in [analytics](/workspace-management/analytics) help detect trends and improve labeling efficiency. ## Still have question? If you ever get stuck or want to explore a feature further, we’re here for you. Just send us a message at . # Labeling Task Types # Span Based **Span labeling** projects allow you to label spans of text within a document. This type is well suited for tasks such as named entity recognition ([NER](/data-studio-projects/nlp-task-types/project-templates#named-entity-recognition)) and part-of-speech ([POS](/data-studio-projects/nlp-task-types/project-templates#part-of-speech)) tagging.
During project creation, you can adjust additional settings as described below. * **Limit selection to a span of 1 token** is useful when you want to ensure that every token in the document is labeled. * **Spans should have at most one label** prevents you from adding multiple labels to a single span. * **Allow arrows to be drawn between labels** lets you draw arrows from one label to another to annotate relationships between words. This is useful for showing that an adjective relates to a noun, or that a pronoun refers to a person. * **Default text** **selection** lets you choose between token-level or character-level selection. Some languages, such as Mandarin, Korean, or Thai, may require character-level selection.
**Note**: If you have already created a project, you can change the configurations through [**Project settings**.](/data-studio-projects/lets-get-labeling#project-settings) {% hint style="info" %} Aside from NER and POS, [audio labeling](/data-studio-projects/nlp-task-types/span-based/audio-project) is another type of spanlabeling task. The main difference is that the audio interface displays a waveform and allows you to modify timestamps. {% endhint %} # OCR Labeling **OCR (Optical Character Recognition) labeling** is part of [span labeling](/data-studio-projects/nlp-task-types/span-based), allowing you to label spans while viewing the original document as a reference. In OCR projects, you can also draw bounding boxes on the media file (`.pdf`, `.tif`, `.jpg`, `.png`, `.gif`, `.docx`, `.pptx`), link these bounding boxes to the transcription text, and view the text and the media file side-by-side. OCR projects are helpful when working with image or document files that contain text. You can perform span labeling tasks like [named entity recognition (NER)](/data-studio-projects/nlp-task-types/project-templates#named-entity-recognition) and [part-of-speech (POS) tagging](/data-studio-projects/nlp-task-types/project-templates#part-of-speech) on the transcribed text. Additionally, you can annotate specific areas of the media files and link it to the corresponding text. ![Overview](/files/ykpHXqp7jYrOccVFEPUs) ### Interface The layout of an OCR project is split into two sections: 1. **Document viewer** - Displays media files on the left. 2. **Text viewer** - Displays the transcription text on the right. You can shrink/enlarge one of the viewers by clicking and dragging the resize handler between the two viewers. You also can hide one of the viewers by clicking one of the arrows in the resize handler. This way, you can hide the document viewer if you wish to focus on labeling the text. ![](/files/kWuAJSRauOIsccBfocR7) You’ll also notice a control panel at the top of the document viewer. This control panel includes: * **Zoom** in and out buttons * **Rotate** counterclockwise or clockwise buttons * **A page indicator** showing the current page number * **Draw** bounding box button ![](/files/mOT9XseNwZN92EkA41ac) ### Labeling in OCR project Applying span labels in the text viewer works the same way as in [span labeling](/data-studio-projects/lets-get-labeling/span-based). What makes an OCR project different is the capability to draw bounding boxes, and link those boxes to spans of text. #### How to link the bounding box to the text? 1. Click the rightmost icon in the control panel to enable drawing mode. The icon turns blue when drawing mode is enabled. ![](/files/8hOnJLMnx6n87ekKYHML) 2. Click and drag in the document viewer to draw a bounding box. Once you release the cursor, the bounding box appears, and a tooltip prompts you to select the corresponding text. ![](/files/8DOenRMCQYdVF5pgzELw) 3. With the bounding box selected, highlight a span of text in the text viewer to link it. ![](/files/tiHgzDT5obeBFF1dfblK) 4. To link the bounding box to different text, right-click the bounding box and select **Edit sentence position**. The box will be highlighted again, and you can select new corresponding text in the text viewer. ![](/files/YK6ks7RVbAMWOznMANeP) 5. To adjust or resize a bounding box, click the bounding box to display the resize handles. Then drag the resize handles to adjust its size. ![](/files/kFUd3e6iJk1JV1ICZpKX) ### Reviewing labels Similar to span labels, bounding boxes can also cause conflicts in reviewer mode. ![](/files/0dhjecqmSvJOpK5TdgE3) * **Conflicting bounding box labels** have **red lines and cannot be resized**. Right-click to accept or reject them. ![](/files/BmON57A89Enayvo9rFyD) * Bounding box labels that have **reached the consensus** have **grey lines and can be resized**. * Bounding box labels **applied by reviewers** have **purple lines and can be resized**. Clicking a conflicting, consensus, or reviewer-applied bounding box will highlight the corresponding text. Likewise, clicking a labeled span of text will highlight the corresponding bounding box. # Audio Project Create and label an audio labeling project in Datasaur ### Project creation
Creating an audio project is simple in Datasaur. All the steps are the same as creating other labeling task types. 1. From the **Projects page**, click **Create project**. 2. Upload both your audio file and its transcription. * Accepted audio formats: `.mp3`, `.flac`, and `.wav` * Accepted transcription formats: `.srt` and `.txt` * You can download example audio and transcription files below. {% file src="/files/KcoYlp3WS2cohl7RHgdW" %} Audio file sample {% endfile %} 3. Make sure the transcription file name matches the audio file name. For example: `SampleFile.mp3` and `SampleFile.srt` . When both files have the same name, the system recognizes them as corresponding files.
4. Continue with the remaining project setup steps: **Preview**, **Labeler's tasks**, **Assignment**, and **Project Settings**. ![The Preview step will only display the transcription. The Labeler's tasks step will only allow span labeling.](/files/p6sq7oqb4LWa6zHHFkwm) ### Audio interface legend ![](/files/54gSdGhU3Q8uiX4wbT1b) ### Label an audio project ![](/files/E30ex9h0C9o8GgQ6ea5r) At the top of the interface, you will find an audio player with timestamps. Below it, you will see the transcript, where you can label spans using the label sets you have added. Between the audio player and the transcript, there is a control panel where you can perform the actions listed below, or [watch this brief video for a visual guide](https://www.loom.com/share/a0078486e0fc4fda8b26f0354e2c2100). * Rewind 10 seconds * Play/Pause * Fast forward 10 seconds * Adjust volume * Use the timestamp field to jump to a specific time * Enter **Create timestamp** mode * Zoom in or out of the audio timestamps * Open audio settings (audio speed and auto-scroll) ![The Control Panel](/files/VkbJivu9115DYXktHolj) The **Create timestamp** button allows you to create a new timestamp and link it to the corresponding text. Select a portion of the audio timeline, then highlight the matching span of tokens in the transcript. The timestamp is now linked to that span of tokens. ![](/files/F87DJwKEpYcOdkWuZY5R)
### Edit sentences in audio project When editing transcriptions, the system adjusts timestamp behavior based on how much the text changes. * If the similarity between the original transcription and the edited version is **above 70%**, the timestamp labels are kept, and you only need to adjust the corresponding text. * If the similarity falls **below 70%**, the timestamp labels may be removed. This happens when the edited text differs significantly from the original. The threshold ensures that timestamp labels remain for minor edits but are removed for more major changes. For example, after editing the sentence in the first line, the timestamp labels will disappear because the similarity between the original and edited sentence is below 70%. {% hint style="info" %} If you upload an empty transcription, placeholder lines with **`-`** will be created and automatically associated with timestamps. Editing the placeholder content will remove the timestamp label, and you will need to create it again manually. {% endhint %} ### Multi-channel audio Audio projects can include files with multiple channels, enabling labeling across different audio sources within the same file. * If the audio file does not contain channels, only the **main audio track** is displayed.
* If an uploaded audio file contains multiple channels, the interface displays: * A **main audio track** (combined view). * Up to **2 individual channels.** Any additional channels in the source file are not shown.
**Timestamp labeling behavior** * **Single-channel audio:** * Timestamp labels can be created **directly on the main audio track.** * Channel-specific labeling is not applicable.
* **Multi-channel audio:** * Timestamp labels can be created **only within individual channels.** * Labels created in channels are **reflected in the main audio track.** * The **main audio track does not support direct timestamp labeling.**
This behavior keeps labeling structured at the channel level while maintaining a unified view in the main audio track. You can use the sample file below to test multi-channel audio. {% file src="/files/VkKAqaUBfDLmtPVz9m2Z" %} # Row Based **Row labeling** allows you to label table data on a row-by-row basis. This is commonly used for classification or data extraction.
Unlike span labeling, you label an entire row of data by answering questions about it. If your data is already in a table format, such as `.xls` or `.csv`, you will be automatically directed to create a row labeling project. # Document Based **Document labeling** allows you to label entire documents at once. This is useful when answering questions about documents, such as `.pdf`, or when classifying images.
# Bounding Box **Bounding box labeling** allows you to label specific areas of an image or document by drawing bounding boxes. This helps you to identify parts of the document or image. You can also add text transcription to each bounding box, which is commonly used for OCR tasks.
### Bounding box interface legend
{% hint style="info" %} For more detailed information on bounding box labeling projects, refer to [this page](/data-studio-projects/lets-get-labeling/bounding-box-labeling). {% endhint %} # Conversational **Conversational** **labeling** allows you to label spans in a text document using a chat-style interface. The interface displays each line as a message with bubbles and avatars, making it easier to work with conversational data. You can label individual spans within messages or label entire messages, which is useful for conversation-level classification. Conversational labeling projects use the same label sets as span labeling projects, so existing label configurations are compatible. ![Conversational Overview](/files/g9SgFFNDK0eXhfHSh95u) ### Create a conversational labeling project 1. In step 1 of project creation, upload a file in a supported format for conversational labeling. You can use the sample `.json` file below to setup the project. 2. In step 3 of project creation, set up your label set by creating it from scratch or by uploading a `.csv` label set file. {% file src="/files/4zldBLeR7cs12eWutJfE" %} {% file src="/files/f7tUDqVwwe4ViS6dHzrf" %} {% hint style="info" %} For more detailed information on conversational labeling projects, refer to [this page](/data-studio-projects/lets-get-labeling/conversational-labeling). {% endhint %} ### Additional settings Conversational labeling projects support the same additional settings available in span labeling projects. * **Limit selection to a span of 1 token** is useful when you want to ensureenforce that every token in the document is be labeled. * **Spans should have at most one label** prevents you from adding multiple labels to a single span. * **Allow arrows to be drawn between labels** lets you draw arrows from one label to another to annotate relationships between words. * **Default text selection** allows you to choose between token-level and character-level selection. ![](/files/CwBXNbcDZKnzRARXI9tu) # Mixed Labeling Mixed labeling lets you combine multiple labeling types within a single project. For example, you can label specific spans of text while also classifying the entire document, making the annotation process more flexible within one project. ## Supported combinations ### Span + document labeling This allows you to label specific spans of text while also classifying the entire document. Span labeling is used to mark specific words or phrases, while document labeling provides an overall classification for the document. **When it’s useful:** Use this when you need to extract key information from a document while also categorizing it based on its content. **Example use case:** In contract analysis, you might highlight specific clauses related to payment terms (span labeling) while classifying the entire contract as an NDA or a Service Agreement (document classification). ![](/files/Hf3BwE5AdUpIw8pG0eyF) ### Span + line labeling {% hint style="info" %} **Line labeling** is not a standalone labeling type and cannot be created on its own. It is a sub-type of span labeling and can only be selected if span labeling is enabled during project creation. {% endhint %} In span labeling, you label specific words or phrases within a line. In line labeling, you label the entire line. **When it’s useful:** When analyzing structured text that requires both detailed and broader classifications. **Example use case:** In chatbot training, you label specific entities such as dates or product names within a sentence (span labeling) while categorizing the entire line as a customer complaint or a product inquiry (line labeling). ![](/files/LwRtg0wSmngB43eJ3vHj) ### Span + line + document Labeling {% hint style="info" %} **Line labeling** is not a standalone labeling type and cannot be created on its own. It is a sub-type of span labeling and can only be selected if span labeling is enabled during project creation. {% endhint %} This approach combines span, line, and document labeling in a single workflow. It allows you to label specific text segments (span labeling), categorize individual lines (line labeling), and classify the entire document (document labeling). **When it’s useful:** When multiple levels of labeling are needed to fully understand the content. **Example use case:** In medical report analysis, you label specific symptoms in a sentence (span labeling), classify the line as a diagnosis or patient history (line labeling), and categorize the document as a radiology or pathology report (document labeling).
### Bounding box + document labeling This allows you to label specific areas in an image or scanned document using bounding boxes, while also classifying the entire document. Bounding boxes mark visual elements or text regions, while document labeling assigns an overall category. **When it’s useful:** When analyzing structured documents where both specific elements and overall classification matter. **Example use case:** In invoice processing, you use bounding boxes to identify key fields such as invoice numbers and totals, while classifying the entire document as an invoice or a receipt. ![](/files/C6gCR2gwLNqUCoUTgOsR) ### Row + document labeling This allows you to label individual rows in a structured dataset (such as a table or spreadsheet) while also classifying the entire document. In row labeling, you answer predefined questions about each row based on its content, while document labeling assigns an overall category to the dataset or file. **When it’s useful:** When working with tabular or semi-structured data where each row is a meaningful unit, and the document also requires a high-level category. **Example use case:** In transaction analysis, you answer questions about each row, such as whether the transaction is valid, a refund, or an anomaly (row labeling), while classifying the entire document as a bank statement or expense report (document labeling).
## How to set up the project 1. Create a new project. 2. In step 3, when selecting the labeling type, selecting a type will display the other labeling types that can be combined. ![](/files/g5o5BGoIXBLrz8Yo7jmG) 3. Select all labeling types you want to use before proceeding to the next step. 4. Complete the remaining steps and create the project. 5. After the project is created, you can start labeling using all selected labeling types. ## Adding project/labeling types to an ongoing project You can add additional project/labeling types to an existing project without needing to recreate it. This enables workflows that combine multiple labeling types even after the project has been launched. #### **Supported additions** The following labeling types can be added to an ongoing project depending on the existing labeling type: | **Existing labeling type** | **Can be added midway** | | -------------------------- | ----------------------- | | Span | Line, Document | | Row | Document | | Document | — | | Bounding Box | Document | | Conversational | Document | #### **How to add a labeling type to an ongoing project** There are two ways to access the project settings: **From the Projects page** 1. Click the three-dot icon beside the project. 2. Select **View project settings**. 3. Open **Task settings**. **From inside the project** 1. Open the project. 2. From the reviewer mode, go to the **File** menu and select **Settings**. 3. Open **Task settings**. Once in Task settings, select the additional labeling type you want to add and configure the label/question set for it. # Project Templates Exploring Datasaur's pre-built project templates Datasaur has project templates that help you get started quickly with preconfigured settings. Let's explore each one. ## Named entity recognition Named entity recognition (NER), also referred to as named entity extraction, is the process of identifying and extracting entities within text. These entities are classified into predefined categories representing real-world concepts, such as people, organizations, and locations.
Named entities are not always single tokens. In the example above, `University of London` is an entity that has three tokens. Multi-token entities are common in NER. {% file src="/files/w2CjJprHOVI559OMuqmF" %} ## Part-of-speech Part-of-speech (POS) tagging is the process of labeling each word in a text with its grammatical role based on the sentence context. Identifying the role of each word helps models understand sentence structure and meaning. You can define custom parts of speech for labeling.
{% hint style="info" %} Note: A common industry standard for **English** **POS tagging** is based on the parts of speech defined by the [Penn Treebank Project](https://www.ling.upenn.edu/courses/Fall_2003/ling001/penn_treebank_pos.html). {% endhint %} {% file src="/files/-MbjYIJ7VYQd0FobOuB9" %} ## Coreference Coreference resolution is the task of identifying all expressions in a text that refer to the same entity. It is widely used in applications such as information extraction, text summarization, question answering, and machine translation.
Coreference can include nouns, noun phrases, proper nouns, and pronouns. For example, `his` is a pronoun that refers to `Sherlock`*,* a noun phrase. Coreference resolution reduces ambiguity in a document by linking related expressions. It typically involves labeling relevant phrases first, then drawing arrows to connect them. {% file src="/files/SwORAeTeV8ss0aZpkKMn" %} ## Dependency Dependency parsing is the task of identifying and labeling grammatical relationships between words in a sentence. These relations consist of a head and a dependent. In the example below `Sherlock` is the subject of the verb `became`.
{% hint style="info" %} For **Dependency** labeling, common industry standards include [Universal Dependencies](https://en.wikipedia.org/wiki/Universal_Dependencies) and the [Stanford typed dependencies Manual](https://nlp.stanford.edu/software/dependencies_manual.pdf). {% endhint %} {% file src="/files/TUFbUBPUXjow4XzEcxSD" %} ## Document labeling Document labeling is the task of classifying and categorizing data. Unlike span labeling, the labeler answers questions about the text instead of labeling specific spans within it. This approach is commonly used for tasks such as sentiment analysis or applying metadata to a document. In Datasaur, document labeling can be done at the row level or the document level. The row-level approach uses the row labeling project type.
In document labeling, you can label images, `.pdf` files, and even `.gif` files.
{% file src="/files/AeQPSXcH95fZsFCNmT6x" %} When creating projects via the **Document labeling** project template, the following settings are applied by default. * Any uploaded questions are set as `required`. * Answer sets can be edited. If a labeler types an answer that doesn't match an existing label, she can click **Add `` as a new answer**. ![](/files/IOrad8CqWFN5i3eTLwXk) ## Optical Character Recognition (OCR) Optical Character Recognition (OCR) is the task of converting text in images or scanned documents into machine-readable text. Common OCR use cases include invoices, receipts, and legal documents. [Watch this Youtube video for instructions on creating an OCR project](https://youtu.be/YsObDLKsewY).\ \ In the video, the user creates an OCR project by uploading the original document with its corresponding transcription as a `.txt` file. Alternatively, your workspace can integrate with an OCR provider of your choice. Datasaur works with any OCR provider you choose. To let us know which provider you want to use, contact us at . Once integrated, the OCR technology will generate transcriptions automatically during project creation, specifically in step 2.
{% hint style="info" %} Note: when uploading OCR document pairs, ensure that each image file and its corresponding transcription have the same file name. For example, `SEC.pdf` and `SEC.txt`. {% endhint %} ## How to use a template 1. Open the Projects page. 2. Click **templates** beside the **Create project** button. 3. Select the desired template to start creating a project from it. 4. You can also open **View all templates** to see the complete list of available built-in templates and templates you have created yourself. ![Project Template](/files/OvP62hC4jR91ZpQG3AlE) ## Create your own template You can create your own reusable project templates by saving an existing project as a template. This allows you to quickly reuse the same label setup, project configuration, and other supported settings for future projects. You can create your own template by following these steps: 1. Create a project with all the desired settings and configurations. 2. On the Projects page, click the triple-dot menu on the project and select **Save as template**. 3. Rename the template as desired and optionally upload an icon. ![](/files/rXMZ3DnvOtFjc7hoTsBL) The template will save supported project configurations from the selected project, including: * Project labeling type. * The project labeling type is saved as part of the template. However, if the uploaded data is incompatible with the labeling type, project creation will still proceed without blocking the user. * Label sets and questions. * Project settings (task settings and administrator settings). * Selected file transformer (if a file transformer was used during project creation). * Tokenizer configuration (for span labeling projects only). * OCR/ASR method configuration (if OCR/ASR was used during project creation). * Enabled extensions. Once saved, templates can help standardize project setup and reduce repetitive configuration work. # Test Project Set a test for your labelers! Lets create a test project for your labelers to assess how well they understand the project guidelines. In a **Test project**, all pre-applied labels or labels applied by reviewers are considered as ground truth tabels. To pass the test, labelers must match their labels to these ground truth labels.
## Create a test project The **Test project** template is only available for team projects and has 4 steps. Any pre-labeled file conforming to the [supported formats](/compatibility-and-updates/supported-formats) can be uploaded in step 1 of project creation. ### **Assign the appropriate labeler** Assign the project to the labelers, then click **Next**.
### **Start labeling the spans** As the project creator, you will start in reviewer mode by default. * If you upload an **unlabeled file**, you can label spans yourself; these labels become the ground truth labels. * If you upload a **pre-labeled file**, you save time, as the ground truth labels are already applied.

Reviewer Mode

{% hint style="info" %} If you upload a pre-labeled file for the **Test project**, the labels **will not** be displayed in the labeler mode. {% endhint %} ### **Labeler has finished the project** Once the labelers click **Mark as complete**, navigate to the reviewer mode. Any conflicts between the labelers' labels and the ground truth labels will be visible.

Reviewer Mode with All Labelers View

By default, you will see all labelers' work. You can view each labeler’s work using the **Show work from** dropdown in the **Test project** extension. When viewing **all labelers**’ work, the color coding is: * **Gray:** Pre-labeled labels. * **Violet:** Ground truth labels applied by the reviewer on reviewer mode. * **Red:** Labels applied by the labelers.

Reviewer Mode with Individual Labelers View

When viewing **an individual labeler's** work, the color coding is: 1. **Gray:** Labels applied by the labeler that match the ground truth labels. 2. **Dark blue:** Labels applied by the labeler that **do** **not** **match** the ground truth labels. 3. **Violet:** Ground truth labels that were not answered by the labeler. To assess labelers' performance, go to the **Test project** extension to see the detailed information about matched labels, incorrect labels, and total labeling time. You can download this information by clicking the triple-dot menu and selecting **Export summary** **(.csv)**.
{% file src="/files/efB4OyLe1ZN5GfZboRLe" %} ## Supported labeling types A **Test project** is available for span labeling, row labeling, and document labeling projects. ## Upload pre-labeled files In a **Test project**, you can upload pre-labeled files as well as unlabeled files. You can upload [pre-labeled files](/data-studio-projects/pre-labeled-project) with the formats listed below. | Task type | Formats | | ---------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Span labeling | [.tsv](https://docs.datasaur.ai/data-studio-projects/nlp-task-types/project-templates/pages/hnBPIPVkJZS5fLhyUjKO#iob-specialized-.tsv), [.tsv\_non\_iob](/compatibility-and-updates/supported-formats#tsv_non_iob), [.json](/compatibility-and-updates/supported-formats#json_tabular), [.json\_advanced](/compatibility-and-updates/supported-formats#json_advanced) | | Span labeling with arrows | [.tsv\_non\_iob](/compatibility-and-updates/supported-formats#tsv_non_iob), [.json\_advanced](/compatibility-and-updates/supported-formats#json_advanced), [.conllu](/compatibility-and-updates/supported-formats#conll-u) | | Span with character-based labeling | [.json\_advanced](/compatibility-and-updates/supported-formats#json_advanced) | | Row labeling | [.csv](/compatibility-and-updates/supported-formats#csv), [.tsv](/compatibility-and-updates/supported-formats#tsv)., [.xlsx](/compatibility-and-updates/supported-formats#xlsx) | | Document labeling | [.csv](/compatibility-and-updates/supported-formats#csv), [.tsv](/compatibility-and-updates/supported-formats#tsv)., [.xlsx](/compatibility-and-updates/supported-formats#xlsx) | # Create a Project There are two ways to create a project: * Create it from scratch by clicking the **Create project** button. * Choose a project template shortcut with preconfigured settings for specific use cases.
In this example, we are going to create a span labeling project. If you would like a tutorial on creating a project for [row labeling](https://www.youtube.com/watch?v=KDRHJ7JVuDk\&t=43s), [audio labeling](https://www.youtube.com/watch?v=9IrY-xcta7w), [OCR labeling](https://www.youtube.com/watch?v=YsObDLKsewY), [bounding box labeling](https://youtu.be/5j4gV3tu1Ps), or [document labeling](https://www.youtube.com/watch?v=eylB9EaxzbI) projects, watch their corresponding Youtube videos. There are five steps to create a project: 1. Upload 2. Preview 3. Labeler's tasks 4. Assignment 5. Project settings ## Step 1: Upload your files
You can see a list of the file formats that are natively supported for each project type by expanding the **Supported file types** section. As an example, we will create a span labeling project by uploading several `.txt` files. {% hint style="info" %} When uploading multiple files, ensure they are all in the same file format. {% endhint %}
The maximum file size allowed is **50 MB**. Files can be uploaded in three ways: * Drag and drop, * Browse files from your computer, * Fetch files from external object storage. {% hint style="info" %} If you are interested in creating project via API, you can find the documentation [here](https://datasaurai.gitbook.io/datasaur/api/create-new-project). {% endhint %} ### Add project tags You can add one or more project tags by selecting existing tags or creating a new one.
If you forget to add tags at this step, you can add them later from the **Projects** page. Follow the steps to [add project tags](https://docs.datasaur.ai/workspace-management/project-management#create-tags) for guidance. ## Step 2: Preview files In this step, you choose how to divide your data into lines and how to tokenize it. ### Line separator The line separator determines how your data are split into separate rows in the labeling interface. There are two native options available: 1. **New line** – Divides your data by paragraph. Each row in the labeling interface will contain one paragraph from your original text. 2. **Dot (.)** – Divides your data by sentence. Each row will contain one sentence from your text. ### Tokenizer The tokenizer determines how your text is broken down into tokens for labeling. There are two native options available: 1. **Wink tokenizer** – separates certain punctuation from words. 2. **Whitespace tokenizer** – splits text at each space, keeping punctuation attached to words.

Wink tokenizer

Whitespace tokenizer

## Step 3: Labeler's tasks In this step, you choose the labeling type you want to work on. A detailed explanation of each task type can be found on the [Labeling Task Types](https://docs.datasaur.ai/nlp-projects/nlp-task-types) page. Since we have previously uploaded `.txt` files, the available task types are span labeling and document labeling. In this example, we will choose span labeling. This means we need to provide labels for use in the project. There are three ways to create or upload labels: 1. **Create label set from scratch**\ Select **Create your own** to manually add your labels. You can also select the color for each label. 2. **Upload label set from a file**\ Drag and drop your `.csv` label set file or select **Browse files** to upload. The CSV format is as follows: 1. Place your first label in cell **A1**. 2. Add subsequent labels down column A (**A2, A3, A4, …**). 3. **Browse from library**\ In your team workspace, the **Label management** page lets you create, edit, and delete label sets. You can reuse saved label sets instead of re-uploading or recreating them for each new project.
### Span labeling settings At the bottom of the page, you'll see a section called **Span labeling** where you can configure several things.
* **Limit selection to a span of 1 token** is useful when you want to enforce that every token in the document must be labeled. * **Spans should have at most one label** does not allow you to add multiple labels to a single span. * **Allow arrows to be drawn between labels** allows you to draw arrows from one label to another to annotate relationships between words. This is useful for showing that an adjective is related to a noun, or a pronoun is referring to a person. * **Default text** **selection** allows you to select whether token or character selection. Some languages may require you to change the selection to character selection, i.e. Mandarin, Korean, or Thai. ## Step 4: Assignment In this step, you assign labelers and reviewers to the project.There are 3 roles available: * **Labeler** * **Labeler & Reviewer** * **Reviewer** Workspace admins have only two options: **Labeler & Reviewer**, and **Reviewer**. Admins always have access to the reviewer mode for any project.
### Conflict resolution The **Peer review consensus** setting determines how many labelers must agree on a label for it to be automatically accepted. The slider allows you to set the threshold for automatic acceptance. For highly sensitive projects where there is no room for error, you can require all assigned labelers to agree. For less sensitive projects where efficiency and cost are more important than accuracy, a majority vote may be enough. Any labels that don’t meet the threshold must be manually reviewed by reviewers or the project creator. If you choose **No consensus**, all labelers’ labels are treated as conflicting. ### Dynamic review assignment Enabling this option automatically assign a reviewer when labelers **have conflicts** in a project. The detailed information can be found in the [Dynamic Review Capabilities](https://datasaurai.gitbook.io/datasaur/getting-started/creating-a-project/dynamic-review-capabilities) page. ## Step 5: Project settings In this step, we chose some final, advanced admin settings for the project. Keep in mind that most of these choices are intended for advanced requirements.
### Labeling settings * **Label set modification** – Allows labelers to add, edit, or remove labels in the project. * **Text modification** – Allows labelers to edit the text of the dataset. Learn more in [Text modification](#text-modification) section. * **Mask Personally Identifiable Information (PII)** – Lets admins mask sensitive information with asterisks or random characters. Learn more in the [Mask Personally Identifiable Information (PII)](#mask-personally-identifiable-information-pii) section. * **Allow marking unapplied label classes as N/A** – Lets labelers mark unused labels as not applicable (N/A). * **Rapid labeling feedback** — Labelers receive ongoing feedback on the status of their submitted labels (e.g., accepted or rejected) while the project is in progress. * **Require approval for pre-labeled labels** — Pre-labeled data must be reviewed and explicitly approved by labelers before it is counted as their work. This setting cannot be changed after the project is created. #### Text modification **Text modification** prevents labelers and reviewers from editing the document text. Refer to the table below for detailed permission.
#### Mask Personally Identifiable Information (PII) After enabling the setting, click **Data masking settings** to define the masking method and the attributes to mask.
The **masking method** determines how the selected information attributes are anonymized. The following masking methods are available: * **Random character**: Replace the personal information with random characters. * Example: `May 23rd, 2022` → `Pgh 78ax, 9928` * **Asterisk (alphanumeric)**: Replace alphanumeric characters in the personal information with asterisks while keeping other characters (such as spaces or symbols). * Example: `John Doe` → `**** ***` * **Asterisk (all characters)**: Replace all characters in the personal information with asterisks, including letters, numbers, spaces, and symbols. * Example: `John@gmail.com` → `**************` * **Field name** *(Row Labeling projects only)*: Replace the personal information with its field or category name. This allows context-aware masking without revealing the original text length. * Example: `John Doe` → `[Name]` To select the information attributes to mask, check or uncheck the **PII fields** in the table. You can also define a **regular expression rule** to specify which entities should be masked. ### Reviewing settings * **Show labeler names in Review Mode** – By default, names are shown. Uncheck this option to hide names and reduce bias. * **Allow reviewers to apply new labels in Review Mode** — When enabled, reviewers can create and apply new labels during review. When disabled, reviewers can only accept, reject, or replace labels created by labelers. * **Note:** Labels from ML-assisted labeling or Data Programming can still be applied. * **Show rejected labels in Review Mode** – Allows reviewers to see labels they have rejected. * **Show labels from inactive label set in Review Mode** – If your project has multiple label sets, this shows labels from all label sets at once. * **Show original sentences in Review Mode** – Shows the original sentences alongisde any edits made by labelers. * **Set notification for labeler's project completion** – By default, reviewers are notified when all labelers mark their work as complete. Use this slider to set the number of completed labelers that will trigger the email notification. Once these settings are configured, click the **Launch project** button to create the project.\ \ **Happy labeling!** # Data Formats Datasaur supports a wide range of data import formats. The available formats depend on the [task type](https://datasaurai.gitbook.io/datasaur/overview/task-type), as described in the table below. Click on any format to see a detailed explanation of the file structure Datasaur expects. {% hint style="info" %} If you don’t see your preferred file format below, you can use [file transformers](https://datasaurai.gitbook.io/datasaur/workspace-management/file-transformer/upload-file-transformer) to upload a custom format. {% endhint %} ## Available formats | [**Span-based**](/data-studio-projects/nlp-task-types/span-based) | [.txt](/compatibility-and-updates/supported-formats#txt), [.tsv](/compatibility-and-updates/supported-formats#iob-specialized-tsv), [.json](/compatibility-and-updates/supported-formats#json) | | -------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | [**Span-based with arrows**](/data-studio-projects/lets-get-labeling/span-based#draw-arrows) | [.txt](/compatibility-and-updates/supported-formats#txt), [.tsv](/compatibility-and-updates/supported-formats#iob-specialized-tsv), [.tsv-non-iob](/compatibility-and-updates/supported-formats#tsv_non_iob), [.json](/compatibility-and-updates/supported-formats#json), [.conllu](/compatibility-and-updates/supported-formats#conll-u) | | [**Span-based with audio**](/data-studio-projects/nlp-task-types/span-based/audio-project) |

Media: .mp3, .m4a, .aac, .flac, .wav
Transcription: .srt, .txt, .vtt, .json

| | [**Span-based with document**](/data-studio-projects/nlp-task-types/document-based) |

Media: .bmp, .doc, .docx, .pdf, .ppt, .pptx, .jpeg, .jpg, .png, .tiff, .tif, .webp

Transcription: .json, .txt, .tsv

| | [**Bounding box labeling**](/data-studio-projects/lets-get-labeling/bounding-box-labeling) | [.bmp](/compatibility-and-updates/supported-formats#bmp), [.gif](https://datasaurai.gitbook.io/datasaur/supported-formats#gif), [.jpeg](/compatibility-and-updates/supported-formats#jpeg-and-jpg), [.jpg](/compatibility-and-updates/supported-formats#jpeg-and-jpg), [.pdf](/compatibility-and-updates/supported-formats#pdf), [.png](/compatibility-and-updates/supported-formats#png), [.svg](/compatibility-and-updates/supported-formats#svg), [.tiff](/compatibility-and-updates/supported-formats#tiff-and-tif), [.tif](/compatibility-and-updates/supported-formats#tiff-and-tif), [.webp](/compatibility-and-updates/supported-formats#webp) | | [**Conversational labeling**](/data-studio-projects/nlp-task-types/conversational) | [.txt](/compatibility-and-updates/supported-formats#txt), [.json](/compatibility-and-updates/supported-formats#json) | | [**Row-based (text classification)**](/data-studio-projects/nlp-task-types/row-based) | [.csv](/compatibility-and-updates/supported-formats#csv), [.json](/compatibility-and-updates/supported-formats#json), [.jsonl](/compatibility-and-updates/supported-formats#jsonl-json-lines), [.tsv](/compatibility-and-updates/supported-formats#tsv), [.txt](/compatibility-and-updates/supported-formats#txt), [.xls](/compatibility-and-updates/supported-formats#xls-and-xlsx), [.xlsx](/compatibility-and-updates/supported-formats#xls-and-xlsx) | | [**Document-based\***](/advanced/extensions/document-and-row-labeling) | [.bmp](/compatibility-and-updates/supported-formats#bmp), [.csv](/compatibility-and-updates/supported-formats#csv), [.gif](/compatibility-and-updates/supported-formats#gif), [.html](/compatibility-and-updates/supported-formats#html), [.jpeg](/compatibility-and-updates/supported-formats#jpeg-and-jpg), [.jpg](/compatibility-and-updates/supported-formats#jpeg-and-jpg), [.json](/compatibility-and-updates/supported-formats#json), [.md](/compatibility-and-updates/supported-formats#md-markdown), [.mp4](/compatibility-and-updates/supported-formats#mp4), [.pdf](/compatibility-and-updates/supported-formats#pdf), [.png](/compatibility-and-updates/supported-formats#png), [.svg](/compatibility-and-updates/supported-formats#svg), [.tiff](/compatibility-and-updates/supported-formats#tiff-and-tif), [.tif](/compatibility-and-updates/supported-formats#tiff-and-tif), [.txt](/compatibility-and-updates/supported-formats#txt), [.tsv](/compatibility-and-updates/supported-formats#iob-specialized-tsv), [.uri](/compatibility-and-updates/supported-formats#url-urls-uri), [.url](/compatibility-and-updates/supported-formats#url-urls-uri), [.urls](/compatibility-and-updates/supported-formats#url), [.webp](/compatibility-and-updates/supported-formats#webp) | | [**LLM Evaluation (fine tuning)**](https://datasaurai.gitbook.io/datasaur/llm-projects/evaluation) | [.csv](https://datasaurai.gitbook.io/datasaur/getting-started/lets-get-labeling/llm-project-type#creating-an-llm-evaluation-project-in-datasaur-a-4-step-guide) | | [**LLM Ranking (RLHF)**](https://datasaurai.gitbook.io/datasaur/llm-projects/ranking-rlhf) | [.csv](https://datasaurai.gitbook.io/datasaur/getting-started/lets-get-labeling/llm-project-type#creating-an-llm-ranking-project-in-datasaur-a-4-step-guide) | ## Size limit * Text-based file: 50 MB per file * Example: [.txt](/compatibility-and-updates/supported-formats#txt), [.tsv](/compatibility-and-updates/supported-formats#iob-specialized-tsv), [.json](/compatibility-and-updates/supported-formats#json), and [.csv](/compatibility-and-updates/supported-formats#csv) * Multimedia & image file: 500 MB per file * Example: [Video](/compatibility-and-updates/supported-formats#mp4), [Image](/compatibility-and-updates/supported-formats#jpeg-and-jpg), [Audio](/compatibility-and-updates/supported-formats#mp3), and [PDF](/compatibility-and-updates/supported-formats#pdf) * Project size: 1.5 GB {% hint style="info" %} To create projects with larger files, use [Robosaur](/integrations/robosaur). For assistance, contact us at **.** {% endhint %} ## Important notes When uploading documents for OCR labeling and audio labeling, make sure each image file and its corresponding transcription file have the same name. For example, `unicef.jpg` and `unicef.txt`. # Data Samples This page provides sample datasets that you can use to create **span labeling, span labeling with arrows, row labeling, bounding box labeling, document labeling** projects and explore the labeling interface.\ \ To create an [audio labeling](https://datasaurai.gitbook.io/datasaur/overview/task-type/span-based/audio-project) project or explore [LLM Labs](https://datasaurai.gitbook.io/datasaur/llm-projects/llm-introduction), visit their respective pages. Each page includes sample data that you can use. ## Span labeling The following `.zip` files include the sample dataset and the label sets. The `.txt` files contain the data to be labeled. Upload the `.txt` files in **step 1** of project creation. The `.csv` files contain the label set (taxonomy) to be applied to the dataset. Upload the `.csv` files in **step 3**. {% file src="/files/bjmQXTdJ9AIyhHojGhRx" %} {% file src="/files/P2dtSkMQi1wfJ0pcHt7u" %} ### Span labeling with arrows The following `.zip` files include the sample dataset and the label sets. The `.tsv` files contain the dataset to be labeled. Upload the `.tsv` files in **step 1** of project creation. The `.csv` files contain the label set (taxonomy) to be applied to the dataset. Upload the `.csv` in **step 3**. {% file src="/files/EkvD9Ok6kuST7ADUjU4v" %} {% file src="/files/JoqZNsofhCVzif4Q1CpK" %} ## Row labeling (textual classification) Upload the `.csv` in **step 1** of project creation. In **step 3,** you can configure your [question set](/data-studio-projects/lets-get-labeling/label-sets#row-based-document-based-labeling) by creating from scratch or by uploading a `.csv` question set file. In this example, the task is sentiment analysis. {% file src="/files/66mQiyVwgHxtfzDemNLr" %} ## Document labeling (document/image classification) The following `.zip` files include images and `.pdf` files to create a document labeling project. In **step 1** of project creation, you can only upload 1 file type per project. You can configure your [question set](/data-studio-projects/lets-get-labeling/label-sets#row-based-document-based-labeling) by creating from scratch or by uploading a `.csv` question set file. {% file src="/files/AeQPSXcH95fZsFCNmT6x" %} {% file src="/files/jW8teLOKuOFnr6pEimXN" %} ## Bounding box labeling The following `.zip` files include the sample images to create bounding box labeling project. In **step 1** of project creation, you can only upload 1 file type per project. In **step 3**, you can configure the label set by uploading a `.csv` [label set (taxonomy) file](/data-studio-projects/lets-get-labeling/label-sets#label-sets) or by creating it from scratch. {% file src="/files/7oMhStTsXP04s1CCXZ8S" %} ## Conversational labeling In **step 1** of project creation, upload the sample `.json` file from the following `.zip` file to create a conversational labeling project. In **step 3**, you can configure your label set by creating it from scratch or by uploading the `.csv` label set file. {% file src="/files/W9e9vvGdE77NWIpqSMG6" %} # Split Files With the **Split files** option, you can divide each file in your project among labelers so they don’t work on the same data. You can also assign specific files to particular labelers.

Split Files

## Sample case Let's say we have a file named `100-rows-reviewed.csv` with 100 rows. Our goal is to divide this file equally between 5 labelers. 1. In step 1 of project creation, upload the files. 2. In step 2, preview the files. 3. In step 3, set the task type and the question set. 4. In **step 4**, choose 5 members as the labelers. 1. If the purpose in this project is **only to speed up** the labeling process and doesn't require the reviewer judgement, set the consensus to 1. 2. If reviewer judgement is required, set the desired consensus accordingly. 5. Select **Advanced**. 6. In the **Advanced** menu, select the **Split files** checkbox. Then enter the number of parts in the **Parts** field. In this example, we set it to 5. The file will be split into 5 parts, and `- Part 1` through `- Part 5` will be added as suffixes to the file names. 7. Select **Distribute files automatically**. The system will distribute the file parts based on the consensus setting. 1. In this case, the file parts will be assigned as follows: * John Doe will get `100-rows-reviewed.csv - Part 1` * Johnson Dominic will get `100-rows-reviewed.csv - Part 2` * James Doe will get `100-rows-reviewed.csv - Part 3` * Jason Doe will get `100-rows-reviewed.csv - Part 4` * Jeremy Doe will get `100-rows-reviewed.csv - Part 5` 2. They will only see their assigned file parts in the project. ## Additional information * The **Split files** option is only available during project creation. File parts cannot be split or merged after the project is created. * Labelers will **only receive** the assigned parts. * Reviewers will receive all file parts. * The **Split files** option is available for span labeling, row labeling, and conversational labeling projects. * The **Distribute files automatically** option is available for span labeling, row labeling, conversational labeling, bounding box labeling, and document labeling projects. # Consensus Allows you to review your project automatically. **Consensus** determines how many labelers must agree before a label is automatically accepted in a project. If a label does not meet the consensus threshold, it is marked as **conflict**.\ \ Currently, two types of consensus are available: 1. **No consensus** – Every label is marked as a conflict and must be resolved by a reviewer. 2. **Peer review consensus** – Labels are automatically accepted once the required number of labelers agree. For example, if the consensus is set to 1, every label will be automatically accepted. ## No consensus To enable **No consensus**, follow these steps: 1. During project creation, go to **Step 4.** 2. In the **Conflict resolution** section, select **No consensus**. ![](/files/6R50B0rliTVVaSauqc38) ### **Use case** The **No consensus** setting can be used if you want your reviewers to review every label applied by the labelers. All applied labels will be considered as conflicts, and reviewers must manually accept or reject them in reviewer mode. ## Peer review consensus **Peer review consensus** is explained in the [QA/Review](/data-studio-projects/reviewing-nlp-projects) section. # Dynamic Review Capabilities Allows you to assign another person to review your project automatically Dynamic review automatically assigns a reviewer when labelers **have conflicts** in a project. This feature works best for small teams with only a few labelers available. The assigned dynamic reviewer will receive an email notification when a conflict occurs in the project. ## How do we enable Dynamic Review? Dynamic review setting is available in step 4 of project creation**.** 1. Assign at least 2 labelers and set the consensus to 2 to allow conflicts to occur and be sent for review. ![](/files/sLLlWdma2TbcOZ5fMF6g) 2. Enable **Allow dynamic review assignment** setting. Choose how to assign the reviewer: 1. **Any other team member** automatically selects one team member, regardless of their role in the workspace, except those who have been assigned as labelers in the project. 2. **Any reviewer** automatically selects one team member whose role in the workspace is **Reviewer**. 3. **Dedicated member** allows you to select a specific member as the reviewer. ![](/files/YpPOjNYUIH7n53XWk83y) ## Two Possible Outcomes: ### No conflicts 1. Admin assign at least 2 labelers. 2. Labelers start labeling. 3. Labelers complete the project and mark it as complete. 4. No conflicts are found. 5. The project status will automatically update as **Completed**. ### Conflicts 1. Admin assign at least 2 labelers. 2. Labelers start labeling. 3. Labelers complete the project and mark it as complete. 4. At least 1 conflict is found. 5. The project status will automatically update to **Ready for review** and the assigned dynamic reviewer will receive a notification. # Pre-Labeled Project **Pre-labeled projects** allow you to create a new labeling project using a file that already contains labeled data. This helps you start faster since you don’t have to label everything from scratch. ## Use cases It is especially useful in the following scenarios: * **Streamlined onboarding:** When starting a new project that shares the same labeling schema as previous one, you can use a pre-labeled file to quickly set up the new project. * **Consistency in labeling:** When you need to apply a standard set of labels across multiple projects, pre-labeled projects help ensure uniformity. * **Efficiency:** Save time by using pre-defined labels for projects with known labeling requirements. * **Data preparation:** Import data with preliminary labels from external sources directly into Datasaur. ## How to create pre-labeled projects ### Span labeling 1. Prepare the pre-labeled file. Supported formats include: * **Span:** [IOB TSV](https://docs.datasaur.ai/data-studio-projects/pages/hnBPIPVkJZS5fLhyUjKO#iob-specialized-.tsv), [TSV non-IOB](/compatibility-and-updates/supported-formats#tsv_non_iob), [JSON Simplified](/compatibility-and-updates/supported-formats#json), [JSON Advanced](/compatibility-and-updates/supported-formats#json_advanced), [CoNLL-U](/compatibility-and-updates/supported-formats#conll-u), [CoNLL 2003](/compatibility-and-updates/supported-formats#conll_2003), [Datasaur Schema (.json)](/compatibility-and-updates/supported-formats#datasaur-schema-format) * **Span with arrows:** [TSV non-IOB](/compatibility-and-updates/supported-formats#tsv_non_iob), [JSON Advanced](/compatibility-and-updates/supported-formats#json_advanced), [CoNLL-U](/compatibility-and-updates/supported-formats#conll-u), [Datasaur Schema (.json)](/compatibility-and-updates/supported-formats#datasaur-schema-format) * **Character-based span labeling:** [JSON Simplified](/compatibility-and-updates/supported-formats#json), [JSON Advanced](/compatibility-and-updates/supported-formats#json_advanced),[Datasaur Schema (.json)](/compatibility-and-updates/supported-formats#datasaur-schema-format) 2. Go to **Projects** page and click **Create project.** 3. In step 1, upload the pre-labeled file.
4. Complete the remaining project setup steps, and your span labeling project will be ready with pre-labeled data.
### Row labeling 1. Prepare the pre-labeled file. Supported formats include: [CSV](/compatibility-and-updates/supported-formats#csv), [JSON Tabular](/compatibility-and-updates/supported-formats#json_tabular), [TSV](/compatibility-and-updates/supported-formats#tsv), [XLS and XLSX](/compatibility-and-updates/supported-formats#xls-and-xlsx), [JSON Lines](/compatibility-and-updates/supported-formats#jsonl-json-lines), [Datasaur Schema (.json)](/compatibility-and-updates/supported-formats#datasaur-schema-format) * Include a column in the file for the answers to the questions you will configure in step 3 of project creation. 2. Go to **Projects** page and click **Create project.** 3. In step 1, upload the pre-labeled file.
4. In step 3, create or upload the question set. Then link each answer column to its question by using [**Refer answer to table column**](/data-studio-projects/lets-get-labeling/label-sets/refer-answer-to-table-column) in **More settings** accordion.
5. Complete the remaining project setup steps, and your row labeling project will be ready with pre-labeled data.
### Document labeling 1. Prepare your documents and pre-labeled answer files. Pre-labeling works as long as the media files are uploaded together with their answer files. * **Notes:** * **Answer file**: A JSON file that contains the answers to your question set. The filename must end with `.answer.json`. Example: ```json { "caption": "A realistic photograph of a white sedan parked on an asphalt road, facing the camera at a front and slightly right angle. The car is centered slightly to the left, with a visible license plate reading 'HZ20 SBV'. The natural daylight provides bright and clear lighting across the scene. In the midground, the asphalt road extends horizontally, flanked by green grassy areas with scattered bushes. The background features a clear, blue sky and a line of four wind turbines with white blades and pale orange towers positioned along a grassy landscape with a body of water visible behind them. The entire composition centers on the car with the wind turbines providing a modern, eco-friendly backdrop." } ``` * **File naming**: Each document and its answer file must have the same filename. For example, if the document is named `a.jpg`, its answer file should be named `a.answer.json`. * **Multiple documents**: If you have multiple documents and multiple answer files, prepare them like this: * `a.jpg`, `a.answer.json` * `b.jpg`, `b.answer.json` * `c.jpg`, `c.answer.json` 2. Go to **Projects** page and click **Create project.** 3. In step 1, upload the documents along with their answer files.
4. In step 3, create or upload the question set. Ensure that the answers in the pre-labeled answer files are configured in this step.
5. Complete the remaining project setup steps, and your document labeling project will be ready with pre-labeled data.
### Bounding box labeling 1. Prepare your documents and pre-labeled answer files. Pre-labeling works as long as the media files are uploaded together with their answer files. * **Answer file format**: Supported file formats include: [YOLO (.txt)](/compatibility-and-updates/supported-formats#yolo-txt), [LabelMe (.xml)](/compatibility-and-updates/supported-formats#labelme-xml), [Pascal VOC (.xml)](/compatibility-and-updates/supported-formats#pascal-voc-xml), [Datasaur Schema (.json)](/compatibility-and-updates/supported-formats#datasaur-schema-format) * **File naming:** Each document and its answer file must have the same filename. For example, if the document file is named `a.jpg`, its answer file in YOLO format should be named `a.txt`. * **Multiple documents**: If you have multiple documents and multiple answer files, prepare them like this: * `a.jpg`, `a.txt` * `b.jpg`, `b.txt` * `c.jpg`, `c.txt` 2. Go to **Projects** page and click **Create project.** 3. In step 1, upload the documents along with their answer files. * **Notes:** * Labels in your pre-labeled file automatically match the labels in your label set. * Extra labels in the file that are not found in the label set will be added as new label classes.
4. In step 3, create or upload the label set. 5. Complete the remaining project setup steps, and your bounding box labeling labeling project will be ready with pre-labeled data.
## Pre-labeled propagation By default, pre-labeled data is treated as the **labeler’s submission**. Once applied in labeler mode, it instantly appears in reviewer mode. Depending on the project’s consensus settings, it may appear as as either *accepted* or conflicted labels. This behavior helps speed up labeling when the pre-labeled data is reliable. If the data is unreliable, turn on the **Require approval for pre-labeled labels** setting to ensure labelers review each label before it is applied. ### Require approval for pre-labeled labels/answers This project setting adds a quality control step for pre-labeled data. It is useful when the pre-labeled source is less reliable or when verification is required before the data can be accepted. Instead of automatically accepting all pre-labeled labels or answers, this option requires labelers to **manually** **accept** or **reject** each one before it moves to the review stage. This ensures that only verified data is used. This setting is useful in the following scenarios: * **Untrusted external data**\ When importing labels from external sources, this setting prevents low-quality or inconsistent labels from automatically appearing in reviewer mode before verification. * **Initial quality verification**\ Allows labelers to validate pre-labeled data before continuing with more complex labeling tasks, ensuring a reliable baseline. * **Assisted labeling**\ Allows labelers to treat pre-labeled labels or answers as suggestions, approving or rejecting them based on project requirements. #### Differences from the default behavior
FeatureDefaultRequire approval
PropagationAutomatic. Pre-labeled data is treated as the labeler’s submission and appears in reviewer mode.Manual approval required. Pre-labeled data remains in the labeler’s copy and does not appear in reviewer mode until the labeler accepts it.
Labeler actionNo action required. Labelers can modify or delete pre-labeled data as needed.Manual review required. Labelers must accept or reject each pre-labeled label and answer.
State in reviewer modeImmediately visible, with color depending on consensus rules.Not visible in reviewer mode. They appear only after the labeler accepts them.
You can configure this setting in the step 5 of project creation, by enabling the **Require approval for pre-labeled labels** toggle. This setting can only be defined during project creation and cannot be changed afterward. ![](/files/ug2NEIlMJw4hiveprq2R) {% hint style="info" %} This setting is currently available for the following project types: * Span labeling. * Row labeling. Support for additional labeling types will be added in future releases. {% endhint %}

Hovering on a pre-labeled span label requiring approval

Hovering on a pre-labeled row answer requiring approval

#### Labeler workflow and outcomes When this setting is enabled, pre-labeled labels and answers appear in each labeler’s view with dashed outlines. Labelers must review them before proceeding. **Accept** When accepted, the pre-labeled data becomes the labeler’s submission. It is marked as accepted and appears in reviewer mode. Labelers can accept pre-labeled labels or answers in the following ways: * **Span labeling** * Right-click a pre-labeled label and select **Accept**. * Apply the same label to the same span. * Use the **Labels** extension to: * Accept a single pre-labeled label. * Accept multiple pre-labeled labels using bulk actions. * **Row labeling** * Select the corresponding answer in the **Row labeling** extension and submit. **Reject** When rejected, the pre-labeled data is removed from the labeler’s view and does not appear in reviewer mode. Labelers can reject pre-labeled labels or answers in the following ways: * **Span labeling** * Right-click a pre-labeled label and select **Reject**. * Use the **Labels** extension to: * Reject a single pre-labeled label. * Reject multiple pre-labeled labels using bulk actions. * **Row labeling** * Leave the answer unselected and submit. This workflow ensures that all pre-labeled data is reviewed before reaching the review stage, improving accuracy and data quality. # Let's Get Labeling! Labeling projects fall under three broad categories: **span-based**, **row-based**, and **document-based**. ## Project settings If you have already created a project, you can change the configurations through **Settings**. You can click the **File** menu and choose **Settings**.
### Span-based projects * **Personalization** tab allows you to adjust the font type and font size, show index bar for labels, and configure labeling behavior. This tab is available to labelers and reviewers.
* **Task settings** tab allows you to configure the span labeling settings. This tab is only available to admins.
* **Assignment** tab allows you manage assigned labelers and reviewers, and update review settings. This is only available for projects created in [Datasaur Teams](/workspace-management/workspace) and only available to admins.
* **Administrator** tab allows you to change the settings related to your project.
### Row-based projects * **Task settings** tab allows you to update the number of rows displayed per page, configure how media should be expanded, and enable markdown parsing.
* **Assignment** tab allows you manage assigned labelers and reviewers, and update review settings. This is only available for projects created in [Datasaur Teams](/workspace-management/workspace) and only available to admins.
* **Administrator** tab allows you to change the settings related to your project.
### Document-based projects Document-based projects have similar settings to row-based projects. The difference is the **Auto mark document as complete** toggle in the **Administrator** tab.
## Roles Datasaur has 5 team roles. You can manage members' roles in the **Members** page. ### Admin You automatically become an admin when you create a team. You can create projects, invite members, assign them as labelers or reviewers, and promote them to admins. You can access all of your team's projects and progress. ### Team reviewer Reviewers role can be assigned at the team or project level. As a team reviewer, you can access and review all projects in the team. ### Project reviewer Project reviewers are assigned in step 4 during project creation. If their workspace role is labelers, they can only access and review the projects assigned to them.
### **Labeler** Labeler is the default role when an admin invites someone to a team. As a labeler, you can only see the projects assigned to you. ### **Labeler + reviewer** If you are a team admin, you can assign yourself as a labeler to a project. Since admins can oversee all projects, you are automatically assigned the **Labeler + reviewer** role when you assign yourself in step 4 of project creation. # Span Based In span labeling projects, you label individual tokens or spans of tokens. This method works well for tasks like [NER](https://datasaurai.gitbook.io/datasaur/overview/task-type/project-templates#named-entity-recognition) and[ POS](https://datasaurai.gitbook.io/datasaur/overview/task-type/project-templates#part-of-speech). To see how span labeling works, [watch this YouTube video](https://youtu.be/s4NhOJzeQLM). ### How to label To label a token or span: * Use your pointer to select a word or a group of words in the dataset. * A list of label classes will appear. * Select a label from the list. ### Keyboard shortcuts When you select a span, a label box with available labels appears, allowing you to choose the label you want. The first 9 label classes show their hotkeys on the right. To apply the label, you can click it or use keyboard shortcuts (`1`–`9`).
### Search for labels If you have many labels, you can search in the label box by typing part of a label name. For example, type `date` then select the label using its hotkey (`1`). The hotkeys follow the order of labels currently shown in the label box, not fixed labels. This means the same key (for example, `1`) may apply to different labels depending on your search.
### Apply multiple labels to the same span You can apply multiple labels to the same span in several ways: 1. **Using your pointer:** After you apply the first label, select the same span again, hold `Shift`, and choose another label. You can keep adding labels as long as you hold `Shift` while selecting them. 2. **Using keyboard shortcuts:** Select the span, use the up and down arrow keys to find a label, then press `Shift` + `Enter`. 3. **Using multiple label selection:** Enable this option in the project settings to apply multiple labels without holding `Shift`. See the next section for details.
### Enable multiple labels selection With **Enable multiple label selection**, you can select multiple labels and apply them to the same token or span without holding `Shift`. This makes the process faster and easier. #### Enable the feature Each labeler has the flexibility to choose a preferred method, whether using keyboard shortcuts or enabling the setting in the interface. You can find the setting under **File menu** > **Settings** > **Personalization**.
{% hint style="info" %} You can access **Personalization** settings after the project is created. Each labeler needs to enable this setting for their own project, it does not apply to other labelers. {% endhint %} Once the setting is enabled, follow these steps to apply multiple labels to a span: 1. Select a span. 2. When the label box appears, select the labels you want to apply. You can choose as many as needed. To select labels, you can: 1. Use keyboard shortcuts (`1`–`9`) 2. Use the up and down arrow keys (`↑` and `↓`) and press `Enter` . 3. Click **Apply labels** to apply all selected labels to the span. 1. You can also can use the `Tab` key to move between labels until you reach **Apply label**, then press `Enter`.
#### Modify applied labels To modify the labels applied to a span: * Click the label you want to change. * Deselect the label you want to change, then select the correct one. * Click **Apply label**.
#### Add a new label classes If the label you need isn’t in your label set, type it to create a new one. You’ll see an option to add the label you type as a new label. Once you add it, it will be available for future use.
#### Apply labels to multiple spans In many cases, several spans share the same label. You can apply a label to multiple spans at once by holding `Ctrl` while selecting them. Let’s say you want to apply the same label to two spans: `Sherlock Holmes` and `Dr. John Watson`. **Case 1: If `Sherlock Holmes` and `Dr. John Watson` don't have any labels applied** 1. Select both spans while pressing `Ctrl`. 2. Choose the labels you want. 3. Click **Apply label**.
**Case 2: If `Sherlock Holmes` and `Dr. John Watson` already have the label `PERSON` and you want to replace it with existing label class** 1. Select both spans while pressing `Ctrl`. * The checkboxes will be reset — the `PERSON` label will not be selected. * The **Reset to mixed labels** button appears but is disabled.
2. To change the labels: 1. Select new labels (for example, `ORGANIZATION` and `BOOK TITLE`) * The **Reset to mixed labels** button becomes enabled. Clicking this will resets the label selection to its previous state.
2. Click **Apply label**. Both `ORGANIZATION` and `BOOK TITLE` labels will be applied to `Sherlock Holmes` and `Dr. John Watson` .
**Case 3: If** `Sherlock Holmes` **and** `Dr. John Watson` **already have the label** `PERSON` **and you want to replace it with a new label that is not in the label set** 1. Select both spans while pressing `Ctrl`. * Checkboxes will be reset — the `PERSON` label will not be selected. * The **Reset to mixed labels** button appears but is disabled. 2. To change the labels: 1. Type a new label (for example, `CHARACTER`) then click **Add new label.** The label will be selected automatically. * The **Reset to mixed labels** button becomes enabled. Clicking this resets the label selection to its previous state.
2. Click **Apply labels.** The `Character` label will be applied to `Sherlock Holmes` and `Dr. John Watson` .
**Notes** * Multiple label selection is only available for tokens and spans, not arrows. * If **Allow arrows to be drawn between labels** is enabled, checkboxes will still appear in the label box, but you won’t be able to select multiple labels for arrows. * If **Spans should have at most one label** is enabled (for example, in [POS](https://simple.wikipedia.org/wiki/Parts_of_speech) use cases), checkboxes will still appear in the label box, but you won’t be able to select multiple labels for arrows. ### Add attribute to a label After applying a label to a span, you can attach an [attribute value](/compatibility-and-updates/common-terminology#labeling-concepts) to it by right-clicking the applied label and selecting **Add attribute**.
The attribute value is displayed: * In the **tooltip** when hovering over the applied label. * In the **editor**, next to the label name. For example:
**Attribute ownership** Each labeler's attribute value is independent and only visible to themselves, labelers cannot see each other's attribute values. If two labelers apply an attribute to the same label on the same span, each value belongs to its respective labeler and does not overwrite the other. **Conflict behavior** A label and its attribute value are treated as a single unit. If two labelers apply the same label to the same span but with different attribute values, the entire unit (label and attribute together) will be treated as a **conflict** in the Reviewer mode. For example, if Labeler 1 applies **`ORG: Microsoft`** and Labeler 2 applies **`ORG: Google`**, they will be considered two different conflicting annotations even though both used the same label. ### Edit sentence You can edit a sentence by double-clicking a line. Note that the sentence is tokenized on the server. To apply your changes, you can: * Click **Save**. * Press `Enter` to save. * Press `Shift` + `Enter` to use spaces as token separator instead of the default tokenizer that Datasaur has.
{% hint style="warning" %} When you make significant edits to a sentence or use `Shift` + `Enter` for tokenization—especially at the beginning or end **where labels are already applied**—some labels may be removed. Take extra care to avoid unintended label loss. {% endhint %} ### Insert new lines To add new lines, right-click a line and select **Insert line above** or **Insert line below**.
### Delete lines To delete a line, right-click the line and select **Delete this line**.
### Delete label You can delete a label in two ways: * Right-click the label and select **Delete label**. * Left-click the label and press `Delete` or `Backspace` on your keyboard.
### Delete sentence labels To delete all applied labels in a line, right-click anywhere in the line and select **Delete labels in this line**.
### Draw arrows To draw arrows between labels: 1. Ensure the admin has enabled **Allow arrows to be drawn between labels**. 2. Apply labels to the spans. 3. Click and hold a label to set it as the source. An arrow will appear. 4. While holding the click, move your cursor to the label you want to set as the destination, then release. {% hint style="info" %} * You can also apply labels to arrows. To do this, double-click an arrow and select a label. * You can reverse arrows, delete arrows, and remove labels from arrows by right-clicking an arrow. {% endhint %}
### Go menu You can move to the desired line via the **Go** menu. * **Go to start:** Takes you to the first line. * **Go to end:** Takes you to the last line. * **Go to line:** Takes you to a specific line. * **Go to next unlabeled span:** Takes you to the next unlabeled span. * **Go to previous unlabeled span:** Takes you to the previous unlabeled span. * **Go to next unlabeled line:** Takes you to the next unlabeled line. * **Go to previous unlabeled** l**ine:** Takes you to the previous unlabeled line. * **Go to next file:** Takes you to the next file. * **Go to previous file:** Takes you to the previous file. * **Go to next conflict:** Takes you to the next conflicted label. * **Go to previous conflict:** Takes you to the previous conflicted label.
### View menu This menu allows user to customize their view. * **Show all lines:** Displays all lines, regardless of whether they have labels. * **Show lines without span labels:** Displays only lines that do not have any span labels. * **Show lines based on label status:** Displays lines filtered by their label status. * For labeler mode: * **Suggested labels**: Labels applied by the automation extension. * **Labels from labelers**: Labels applied manually by labelers.

View menu for labeler

* For reviewer mode: * **Consensus labels**: Labels agreed upon by labelers. * **Conflicted labels**: Labels with disagreements between labelers. * **Suggested labels**: Labels applied by the automation extension. * **Labels from reviewers**: Labels applied manually by reviewers. * **Rejected labels**: Labels that have been rejected by reviewers.

View menu for reviewer

Here are previews when the user chooses: * **Show lines without span labels**
* **Show lines based on label status** → **Labels from labelers**.
#### Additional options for Span+Line labeling If the project type is **Span+Line** or **Span+Line+Document labeling**, an additional option **Show lines without line answers** is available in both labeler and reviewer modes. This option can be accessed from **Show unlabeled lines only** → **Lines without line answers**.
#### Additional options for audio labeling If the project type is **Audio labeling**, an additional option called **Show timestamps in text editor** is available in both Labeler and Reviewer modes. When enabled, timestamps are always displayed in the text editor, helping users identify which parts of the transcript are connected to timestamps in the audio. Clicking a timestamp allows users to navigate directly to the corresponding timestamp location in the audio.
### Sort lines by metadata fields In span labeling projects, lines are shown in their original order, which may not be ideal for structured data. Users can define custom metadata during file upload, such as date, user ID, or record ID, and use these fields to sort lines. This helps organize data chronologically, group related records, or follow sequences such as conversation turns. Sorting only affects the current view and does not change the underlying data or other users’ views. Notes about sorting behavior: * Sorting affects only the current user’s view. * Multiple metadata keys can be used, with sorting applied from top to bottom by priority. * Numeric values are sorted numerically, while values containing letters are sorted alphabetically. * Lines with missing or empty metadata values appear after those with valid entries. * The original data remains unchanged. To sort lines by metadata, follow these steps: 1. Open the **View** menu. ![](/files/H5ybqZH9GaYOXahA8mvA) 2. Select one or more metadata keys to sort by. 3. Choose ascending or descending order for each key. 4. Add additional sort conditions if needed. 5. Reorder the conditions by dragging them to set the sorting priority. 6. Close the panel to apply the sorting immediately. 7. To return to the original order, clear all sort conditions. ![](/files/g0dbl3Rg0uOABAJzwq4W) ### Personalization settings This setting allows users to customize their labeling experience. It is accessible from **File** → **Settings** → **Personalization**.
#### Automatically jump to next document when marking as complete When this setting is enabled, marking a document as complete automatically moves you to the next document. You can mark a document as complete from the **List of files** extension or by using a shortcut (`Ctrl + M`). This setting eliminates the need to manually navigate between documents after marking one as complete. #### Show index bar for labels When your project involves extensive paragraph or sentence labeling for longer or multi-line text, you can enable **Show index bar for labels** and disable **Always show labels** to reduce clutter.
This setting is accessible from **File** → **Settings** → **Personalization**.
### Character selection **Character selection** allows you to apply labels at the character level, giving you more precise control than selecting entire words. To enable it: * Go to **File** → **Settings** → **Task settings.** * Open **Default text selection**. * Choose **Character selection.**

Task settings will show in the personal workspace and in the reviewer mode under the team workspace.

You can label characters in two ways: * Select the desired character using your mouse. * Select the character using keyboard shortcuts `Shift` + `→` (Right arrow). ### Highlight an entire sentence If you want to label the entire sentence, click the line number.
### Select multiple lines at once To select multiple lines at once, hold `Shift` and click the desired line numbers. {% hint style="info" %} You can also select multiple lines starting from any line number, for example, lines 4-8. {% endhint %}
### Adjust span selection This feature allows you to modify the selection of an already applied span label without needing to delete and reapply it. To adjust a span, you must have permission to modify the label. To adjust a span selection: 1. Right-click the label you want to adjust. 2. Select **Adjust span selection**. In reviewer mode, if the label is still conflicted and unresolved, this option is not available.
3. You can modify the selection by dragging the selection handles to the desired position. 1. Shortcut, extension, and title bar functionality are disabled while in this mode. 2. All span labeling settings in **Task settings** (such as **Spans should have at most one label** and **Limit selection to a span of 1 token**) still apply.
4. To save changes, click anywhere outside the selection. A saving indicator appears below the selection and disappears once the save is complete.
### Drawing bounding boxes in the OCR interface In [OCR](/data-studio-projects/nlp-task-types/project-templates#optical-character-recognition-ocr) projects, you can draw bounding boxes in the viewer and link them to the corresponding text. To start drawing bounding boxes, click the **Draw bounding box** icon in the top-right corner of the document viewer.
After you create a bounding box on the document viewer, a tooltip appears prompting you to select the text on the right side of the screen.
Select the corresponding text in the text viewer to link it to the bounding box you just created.
### Auto-scroll in the OCR project After you create bounding boxes and link them to the text: * Clicking a bounding box on a page automatically scrolls the text viewer to the corresponding text. * Clicking a span of text or a label automatically scrolls the document viewer to the corresponding bounding box. This feature is especially helpful for PDFs with multiple pages. ### Synchronized scrolling **Synchronized scrolling** synchronizes the scroll position between the document viewer and the text viewer, reducing the need to scroll through both manually and helping keep them aligned. {% hint style="info" %} This feature is currently available only for OCR labeling projects with PDF files. {% endhint %} {% embed url="" %} #### Enable synchronized scrolling Follow these steps to enable this feature: 1. On the bottom bar of an OCR project, click the button to toggle synchronized scrolling. ![](/files/mOT9XseNwZN92EkA41ac) 2. Wait for the mapping process to complete. * The mapping process applies only **to the currently opened document**. To process other documents, open each one individually. * You can check the mapping progress by hovering over the button or viewing the progress indicator at the top of the editor. ![](/files/IKRJknAFbnPl8JXlaTmO) * While the mapping is in progress, you can continue labeling. However, actions that modify the text (such as editing sentences) are disabled. 3. Once the mapping process is complete, a success message appears. ![](/files/pA0amqiDwtqL7StyRtBc) 4. You can then scroll in one viewer, and the other viewer will automatically scroll in sync. {% hint style="info" %} The mapping result is saved for all users assigned to the document. Only one assignee needs to complete the mapping process; others can enable synchronized scrolling without waiting. {% endhint %} #### Trigger auto-scroll To trigger auto-scroll, scroll in the transcription or document viewer using your mouse wheel, touchpad, or the `↓` and `↑` arrow keys. You can also click a span in the transcription to trigger the auto-scroll. #### Disable the feature To temporarily disable this feature, click the **Synchronized Scrolling** button again at the bottom of the page. You can re-enable it at any time. #### Get the best experience To create scroll points between PDF and the transcription, the system maps text using text matching. To improve mapping accuracy: 1. Use native PDFs instead of scanned PDFs. Native PDFs contain embedded text that can be extracted, while scanned PDFs contain images of text that cannot be processed. 2. Avoid having a single line in the transcription cover multiple lines in the PDF. Ideally, each line in the PDF should match one line in the transcription. 3. For best results, set the document viewer zoom to 100%. #### Limitations 1. Enabling PII anonymization may disrupt the mapping process, as masked text can prevent accurate matching. 2. Synchronized scrolling is temporarily disabled when the document is rotated and is re-enabled when it returns to its original orientation. 3. Modifying the transcription (editing, inserting, or deleting sentences) may cause unexpected scrolling behavior, as mapping is based on the original transcription and does not account for later changes. ### Mark as complete Once you have finished labeling, set the **Labeling in progress** status to **Project completed**. This signals to your team that the project is complete and ready for review or export.
# Span + Line Labeling This guide will walk you through how to set up the combination of span + line labeling in one project. ### Create a project 1. From the **Projects** page, click **Create project**. 2. Upload files with supported file types for span labeling. 3. In step 3, select **Span labeling** and **Line labeling**. ![](/files/1A8v5v6kHaOgITYFZiir) 4. Create a label set for span labeling and a question set for line labeling. 5. Complete the remaining steps and create the project. 6. Start labeling by selecting spans or clicking the line number to label the corresponding line. ### **Submit line answers** 1. Enable the **Line labeling** extension. Click the gear icon in the extension sidebar on the right to open the **Manage extensions** dialog and enable it. 2. Right-click the line you want to answer. 3. Select **Answer this line** from the context menu. 4. Enter your answer in the **Line labeling** extension and click **Submit answers**. ![](/files/c529J0WN76f1ZTMxxlGO) ### Get the latest label updates in reviewer mode As a reviewer, you can refresh both span labels and line answers to ensure the latest changes are reflected. When reviewing labeling work, the displayed labels and answers may not be up to date. Open the **Review** extension and click **Update now** to load the latest changes. # Span + Line + Document Labeling This guide will walk you through how to set up the combination of span + line + document Labeling in one project. ### Create a project 1. From the **Projects** page, click **Create project**. 2. Upload files with supported file types for span labeling. 3. In step 3, select **Span labeling**, **Line labeling**, and **Document labeling**. ![](/files/Zpq6u7CPMkp559lFz84f) 4. Create a label set for span labeling, and question sets for line labeling and document labeling. 5. Complete the remaining steps and create the project. ### Working on a span + line + document labeling project When you open the project, **Labels**, **Line labeling**, and **Document labeling** extensions are enabled automatically. * To label a span, highlight the text and choose a label from the label box. * To answer a line, right-click the line and select **Answer this line** from the context menu. Enter your answer in the **Line labeling** extension and click **Submit answers**. * To label a document, open the **Document labeling** extension and answer the questions. ![](/files/iRjKTBS0TYIT5RaZKL7V) ### **Get the latest label updates in reviewer mode** As a reviewer, you can refresh span labels, line answers, and document answers to ensure the latest changes are reflected. When reviewing labeling work, the displayed labels and answers may not be up to date. Open the **Review** extension and click **Update now** to load the latest changes. # Row & Document Based In row labeling and document labeling projects, the goal is to answer questions. In row labeling, you answer questions for each row using the **Row labeling** extension. In document labeling, you answer questions for the entire document using the **Document labeling** extension. You can navigate to the next question by using your mouse or using `Tab` on the keyboard.
### Go menu You can move to the desired row via the **Go** menu. * **Go to start:** Takes you to the first row. * **Go to end:** Takes you to the last row. * **Go to line:** Takes you to a specific row. * **Go to next unlabeled line:** Takes you to the next unlabeled row. * **Go to previous unlabeled** l**ine:** Takes you to the previous unlabeled row. * **Go to next file:** Takes you to the next file. * **Go to previous file:** Takes you to the previous file.
### Row page navigation For row labeling projects, you can choose how many rows are shown per page during project creation. Go to step 2, and set the value in **Number of rows displayed per page** field. By default, all rows are shown on one page.
Inside the project, you can navigate between pages using the pagination control at the top right of the table.
There are three views in row labeling projects: * **Tabular view** (default): Shows the data in a table format.
* **URL view (image)**: Preview the image URLs so you can label based on the image.
* **URL view (website)**: Preview the web page URLs so you can label based on the page content.
### Document navigation To navigate to the next document, use the navigation control located at the bottom left of the labeling interface.
### Required question The asterisk (\*) next to the question indicates that the question requires an answer. Leaving it blank will trigger an error.
### Sort and filter column If you create questions with text field, text area, dropdown, hierarchical dropdown, date, time, checkbox, slider, grouped attributes and URL question types, you can sort and filter the columns. For **text field**, **URL**, and **text area** answer columns, you can filter by searching keywords.
For the **dropdown** and **hierarchical** **dropdown** column, you can filter based on the dropdown value.
For the **date** and **time** columns, you can filter by selecting a date range or a time range.
For the **checkbox** column, you can filter it by true or false to check whether is it checked or not.
For the **slider** column, you can filter it by a specific value or a value range.
For the **grouped attributes** column, click the arrow next to the question type header name to open the filter options.
It will display a list of each question type in the grouped attributes question.
You can start filtering based on the labeled answers by clicking a question and applying the filter.
If filters are already applied, the arrow next to the question header changes into a filter icon.
{% hint style="info" %} Filtering is not supported for radio questions inside the grouped attributes question type. {% endhint %} ### Keyboard shortcuts for dropdown questions When the question type is **dropdown**, keyboard shortcuts are shown in the answer options. In the example below, you can press `1` on your keyboard to select `Children` as the answer.
### Filter rows In reviewer mode of row labeling projects, you can filter rows from the **View** menu. #### Filter unlabeled rows only This filter shows only rows that have not been labeled yet. It helps you quickly find and prioritize data that still needs labeling.
#### Filter conflicted rows only This filter shows only rows with conflicting labels. It helps you identify and review disagreements in the data so you can resolve them.
#### Filter unreviewed rows only This filter shows rows that have not been reviewed yet. It helps reviewers track which data still needs review and ensure everything is checked before finalizing.
{% hint style="info" %} **Unreviewed rows** include both consensus and conflicted rows where the reviewer has not submitted an answer. {% endhint %} #### Filter reviewed rows only This filter shows rows that have already been reviewed and approved. It helps you work only with verified and reliable labeled data.
### Hide and rename headers You can hide and rename headers by right-clicking on a column header. * **Hide column:** The column is hidden only for you. Other labelers are not affected. * **Hide column from labelers:** The column is hidden for all assigned labelers, but remain visible to reviewers. This setting is only available in reviewer mode.
### Insert new rows Inserted rows work like regular rows. They are automatically numbered and can be edited at any time. Both labelers and reviewers can insert new rows in a project. * **Labelers** can insert rows while labeling. * **Reviewers** can insert rows during review, and also decide whether to accept or reject insertions made by labelers. To insert new rows: * Right-click any row. * Select **Insert row above** or **Insert row below.**
### Delete rows Both labelers and reviewers can delete rows in a project. * **Labelers** can delete rows while labeling. * **Reviewers** can delete rows during review, and also decide whether to accept or reject deletions made by labelers. {% hint style="info" %} **Note:** Deleted rows can't be undone and won't automatically replace the original ones. Reviewers decide which version to keep. {% endhint %}
### Mark as complete Once you finish labeling, mark the project as complete by changing the **Labeling in progress** status to **Project completed**. This signals that the project is ready for review or export.
### Personalization settings This setting lets you customize your labeling experience. You can access it from **File** → **Settings** → **Personalization**.
#### Automatically jump to next document when marking as complete When this setting is enabled, marking a document as complete automatically moves you to the next document. You can mark a document as complete from the **List of files** extension or by using a shortcut (`Ctrl + M`). This setting eliminates the need to manually navigate between documents after marking one as complete. ### Row-based with URL view You can label images by adding image URLs in a column of a row-based file. **Prepare your file** * Store images in an accessible location. * Create a row-based file that includes a URL column. * Add the image links to the URL column. * You can add more columns for additional image information. This data cannot be edited later.
**Set the preview** Choose how images are shown in the labeling interface: * **Don't expand:** No image preview is shown. * **Thumbnail:** Shows a small image preview. * **Large:** Shows a larger image preview.
**Set the viewer to URL view** If the main focus of your labeling is the content from URLs, use URL view. 1. During project creation, in step 3, change the viewer setting from **Tabular view** to **URL view**. 2. Set the **URL column** to the column in your row-based file that contains the URLs.
**Start labeling** Inside the project, image previews are shown from the URLs in your row-based file. You can start labeling from the **Row labeling** extension.
**View other columns** Additional information from other columns in your row-based file is available in the **Row labeling** extension.
### Apply answers for multiple rows You can apply the same answer to multiple rows at once. This is useful when multiple rows share the same pattern. There are two ways to select rows: * Use the checkbox in each row. * Hold `Ctrl` on your keyboard to select multiple rows. After selecting the rows, navigate to the **Row labeling** extension to answer the questions. Click **Submit** to apply the answer to all selected rows.
#### Possible cases * Select multiple rows with no answers, then answer all questions. * All selected rows will have the same answers. * Select rows where some have answers and some do not, then answer all questions. * All selected rows will have the same answers. * Select rows where some have answers and some do not, then only answer one question and leave the rest blank. * Only that question will be updated for all selected rows. * If you select multiple rows, questions may show a **Mixed** value when answers differ across rows.
* If you change the value and submit, it will override all answers for the selected rows. * A **Reset** button appears if you want to revert back to the **Mixed** value. {% hint style="info" %} * This feature is only available in the **labeler mode**. * It is not supported for row labeling projects with: * Number of rows per page: 1. * URL viewer enabled. {% endhint %} ### Auto-saved answers Auto-save stores your answers as a draft before you submit them. The draft is kept even if you refresh or close the page. This is useful for projects with many questions.
Auto-saved answers apply to both labeler and reviewer. In reviewer mode, if another reviewer submits an answer in the extension, the current draft is replaced with the latest answers (based on the timestamps). You can also discard drafts by clicking the **three-dot** menu in the extension.
#### Notes and limitations * Auto-saved answers are only available for single row selection. This also applies when discarding drafts. * In row labeling projects, drafts are shown only in the **Row labeling** extension, not in the table. Answers appear in the table only after they are submitted. * Changing the question set will remove all auto-saved answers.
# Bounding Box Labeling ## Bounding box projects This guide shows how to create a bounding box project and how to draw and label bounding boxes on images, PDFs, or documents. You can also [watch this 2:00 minute Youtube video](https://youtu.be/5j4gV3tu1Ps) for a walkthrough. ## Interface legend If you already have a project, use this legend to understand the basics of the labeling interface. Also, see [this :43 second Youtube video](https://youtu.be/H1B_G_t5bXc) to learn how to draw and label bounding boxes.
## Create a project To create a bounding box labeling project: 1. Go to the **Projects** page and click **Create project.**
2. Upload your image or document file. Supported file types: 1. **Images:** `.bmp`, `.gif`, `.jpeg`, `.jpg`, `.png`, `.svg`, `.tiff`, `.tif`, `.webp` 2. **Document**: `.pdf` Here is an example: {% file src="/files/AVeUflb9SqHgUTpk3e7G" %}
3. In step 2, set **Apply OCR Method** to **None**. 4. In step 3, select **Bounding box labeling** and add a label set. 1. Supported formats: `.csv`, `.tsv`, or `.json` 1. For `.csv` and `.tsv`, we support **color names** (example: `red`), **hex values** (example: `#00FF00`), and **RGB** (example: `rgb(0,0,255)`). You can also provide label names only. Other fields such as `captionAllowed` and `captionRequired` will use default values. 2. For `.json`, we support **hex** and **RGB** only. {% file src="/files/88e4WXaWyWBV7b9u2NRd" %} {% file src="/files/TXH3IJwCMJuq4vhBKCiH" %} {% file src="/files/IDatnggwXcViBmOVnXxs" %} {% file src="/files/iG5wYdtJYSKSIkIjYr1E" %} {% file src="/files/idpJJxOEejnkoMFE4BO3" %} 5. Configure the **Allow text caption** setting. It allows labelers to add text to a bounding box.
1. You can type them manually or use an OCR provider. 2. You can also require text for specific labels by enabling the **Require caption** option.
Finish the remaining steps as you would for other project types: **Assignment** and **Project settings**. ## Upload pre-labeled files You can upload pre-labeled files in a bounding box labeling project by uploading a JSON file together with the document you want to label. he JSON file must follow the [Datasaur Schema](/compatibility-and-updates/supported-formats#datasaur-schema-format) format. ## Label sets capabilities ### Label set modification You can update the label set at any time: * **Add a label set**: Drag and drop a file, or use the three-dot menu.
* **Edit a label set**: Open the three-dot menu and select **Edit label set.**
* **Replace a label set**: Open the three-dot menu and select **Replace existing label set**, then create a new one or upload a new file.
By default, both reviewers and labelers can perform these actions. To restrict access, go to **File** > **Settings** > **Administrator** and disable this option.
### Custom attributes for label classes Each label class in the label set can have multiple custom attributes (questions). The supported question types are [Text](https://docs.datasaur.ai/data-studio-projects/lets-get-labeling/pages/sjMjHjQsHtR5BrPf3PUF#id-1.-text-field) and [Dropdown](https://docs.datasaur.ai/data-studio-projects/lets-get-labeling/pages/sjMjHjQsHtR5BrPf3PUF#id-3.-dropdown). Each question can also have a default answer. To modify the custom attributes: 1. Open the label set editor via **Add new label set**, **Edit label set**, or **Replace existing label set**. ![](/files/msIU413fzvgu4B4kZIqB) 2. Go to the **JSON** tab. ![](/files/nNzfxyHjIcP39ByYbeWZ) 3. Add or modify the `questions` field as needed for each label class. You can't modify the `type` and `internalId` of existing questions, as it may affect existing labels. Question structure: 1. **id**: Unique question ID. 2. **label**: Question name. 3. **type**: `TEXT` or `DROPDOWN`. 4. **config**: Question settings 1. **required**: Determines whether the question must be answered. It can be `true` or `false`. 2. **defaultValue**: The default answer for the question 1. For `TEXT` questions, it can be any text. 2. For `DROPDOWN` questions, it must match an `options.id`. 3. **multiline**: Specifies if the `TEXT` question allows multiple lines. It can be `true` or `false`. 4. **options**: An array of options for the `DROPDOWN` question, each containing: 1. **id**: ID of the dropdown option. It can be used in the `defaultValue`. 2. **label**: The text label for the dropdown option. 5. **internalId**: Read-only unique ID used by the system. For a full example, refer to the [Label Sets](#label-sets) section to see a sample bounding box label set with custom attributes. ## Create bounding boxes In bounding box labeling projects, the document viewer displays the uploaded documents or images and allows you to create bounding boxes. There are two cursor types for creating bounding boxes: **Bounding box** and **Text selection**. ### Bounding box Use **Bounding box** cursor to draw a bounding box directly on the document. This method gives you precise control over placement.
### Text selection ![](/files/mwPY7Gem2lAnpOjQcVng) Use **Text selection** to create bounding boxes by selecting text instead of drawing manually. This is faster when working with text-based documents. {% hint style="info" %} * Available only for PDF documents. * Works only with PDFs that contain selectable text (text layers). * PDFs created from tools like Microsoft Word usually contain text layers. Scanned or image-based PDFs typically do not. {% endhint %} To use **Text selection** mode: 1. Enable **Text selection** from the toolbar. 2. Select text by clicking and dragging, or double-click a word. ![](/files/FtSXS9k8eWa3JTO7bTbo) 3. When you release the mouse, a bounding box is created automatically. A label dialog appears so you can assign a label. The selected text is added to the caption field by default. ![](/files/xLOnzaJdD8meUsg0ZKyb) ## Apply labels, captions, and custom attributes After drawing a bounding box, a modal appears with the following fields: * Label, * Caption, * Custom attributes (if configured). Select a label, enter a caption, and answer any attributes as needed. To submit, click anywhere in the document viewer.
To review a bounding box, hover over it to see a tooltip with its values.
### Automatically fill the caption using OCR You can automatically fill captions using OCR (Tesseract). When you draw a bounding box around text, the detected text is added to the **Caption** field. {% hint style="info" %} Processing may take longer for larger text selections. {% endhint %} To enable this setting: 1. In step 3 of project creation, enable **Allow text caption**. 2. Select **Tesseract** from the **OCR provider** dropdown.
After creating the project, you can enable or disable auto-caption from the three-dot menu in the **Bounding box labels** extension. Changes apply to all documents.
**Notes on auto-caption:** 1. It runs only when drawing a bounding box (not after it is created). 2. It runs again if the box is resized while still in the drawing state. 3. It can be canceled using the **(x)** in the **Caption** field. ![](/files/akxY7OS7zD8ev8KAvt6A) 4. It stops if you click outside the modal before the process is finished. In this case: 1. It returns an empty value if the caption is optional. 2. It shows an error if the caption is required. 5. It supports multiple bounding boxes. Captions are separated by line breaks. ## Bounding box arrows / relationships Bounding box relationships allow you to define connections (arrows) between bounding boxes within the same document. This is useful for capturing structural or semantic links such as key–value pairs, question–answer pairs, or grouped elements. In many cases, related information is captured as separate bounding boxes but needs to be understood together. For example: * A field label like “Invoice Number” is associated with its corresponding value. * A “Total Amount” label is linked to the numeric value shown elsewhere on the page. * Multiple cells in a row belong to the same table record. Bounding box relationships enable you to explicitly define these connections, making the data more structured and easier to use for downstream extraction or model training.
**Creating a relationship** — To create a relationship between bounding boxes: 1. Select the **arrow** cursor type at the top of the document viewer. 2. Click the origin bounding box, then click the destination bounding box. 3. (Optional) To add a label, click the arrow and select a label. **Deleting relationships** — To remove a relationship: 1. Right-click on an arrow to open the context menu. 2. Click **Delete.** {% hint style="info" %} * Relationships can only be created between bounding boxes within the same document. * Deleting a bounding box will also remove its associated relationships. {% endhint %} ## Single vertex edit A bounding box is initially drawn as a rectangle. To improve accuracy for document parts that are not perfectly square, you can adjust individual cornet points by holding the `Alt` key (or the `Option` key on Mac) and dragging a corner of the box. ## Move and edit bounding boxes on hover The **Show resize handles on hover** setting allows you to edit bounding boxes by hovering over a box instead of clicking it first. Only admins can configure this setting. ![](/files/ViFFaPxIikGvanaHAZf7) ### Enable the setting 1. During project creation, go to **Step 5** and enable the **Show resize handles on hover** setting. ![](/files/lNWdmkZ9srLHEvkZHumD) 2. To enable it for an existing project, go to **File** → **Settings** → **Administrator** and enable the **Show resize handles on hover** setting. ![](/files/blRXR8jvbxDn4DWH9wZB) ## Manage bounding boxes ### Copy and paste {% hint style="info" %} Bounding boxes can only be copied and pasted within the same document. {% endhint %} You can copy and paste bounding boxes in two ways: * Right-click a bounding box, and select **Copy** or **Paste**. * Select a bounding box, then press `Ctrl`+`C` / `Ctrl`+`V` (or `Command`+`C` and `Command`+`V` on Mac). ### Delete You can delete bounding boxes in two ways: * Right-click a bounding box and select **Delete.** * Select a bounding box, then press `Del`. ### Multiple selection Multiple selection is helpful for copying, pasting, or deleting multiple bounding boxes. To select multiple bounding boxes, hold `Ctrl` while selecting each one. ### Merge multiple bounding boxes You can merge multiple bounding boxes into a single label with multiple connected boxes. The result depends on the label classes: * If all selected bounding boxes have the same label class, they are merged under that label class. * If they have different label classes, the label box appears so you can choose a label class and update the caption before saving. The merged caption is determined by the selection order. For example, if the first box is `Merchant` and the second is `name`, the result will be `Merchant name`. To merge bounding boxes: 1. Select multiple bounding box. 2. Right-click one of the selected boxes and select **Merge.** ### Manage visibility: Show and hide When working with datasets containing many overlapping bounding boxes, the view can become cluttered. You can manage visibility by hiding or showing bounding boxes by label class or individually, helping you focus on specific areas. #### **Hide and show bounding boxes** You can hide and show bounding boxes in three ways: 1. **By label class** 1. Go to the **Bounding box labels** extension. 2. For each label class, there's an eye icon located next to the label instance counter.
3. Click the eye icon to hide all bounding boxes associated with that label class. Click it again to show them. 4. In the extension, you can also see the number of bounding boxes that are currently shown and hidden. 2. **From document viewer, individually** 1. Right-click on a bounding box. 2. Select **Hide**. ![](/files/uJVV37odhMLclbH3ElLI) 3. **From extension, individually** 1. Navigate to the **Bounding box labels** extension. 2. Click the instance counter for a label class. It will show all boxes associated with that label class. 3. Click the eye icon to hide the bounding box. Click it again to show them.
{% hint style="info" %} * If a bounding box is added or updated after you hide it, it will still be visible. This ensures that new or updated data is always shown for review. * Refreshing the page resets all visibility filters and shows all bounding boxes. {% endhint %} ## Reviewer capabilities overview #### Capabilities * View all labelers' work. * Note: Bounding box labels applied in similar positions may overlap. * Apply labels. * Update labels applied by labelers. #### Limitations * Cannot accept or reject labels. * Cannot view all labelers' work with information on who applied each label. * When updating a label applied by a labeler, there is no information or indicator showing that the label has been changed. ## Upload pre-labeled files In bounding box labeling projects, you can upload pre-labeled files. Learn more about creating pre-labeled bounding box labeling projects in [this documentation](https://docs.datasaur.ai/data-studio-projects/pre-labeled-project#bounding-box-labeling). # Conversational Labeling The conversational labeling project is similar to span labeling project, allowing you to apply span labels and draw arrows as usual. See [span labeling](/data-studio-projects/lets-get-labeling/span-based) for more details. Unlike span labeling, conversational labeling focuses more on conversation data and includes message-level labeling. ### Apply message labels To apply a message label, click on the add (**+**) button at the top right of a message.
Select the label class you want to apply to the message.
### Change message labels To change a previously applied label, left-click the label, then select the new label class that you want to apply.
### Delete message labels To delete a message label, right-click on it and select **Delete label**.
# Label Sets / Question Sets ## Span labeling For span labeling, a label set is a single-column `.csv` file that follows the structure below. | Column 1 | | -------- | | Label 1 | | Label 2 | | Label 3 | | etc... | {% file src="/files/RujHf5DMkmhtr7gD4uSg" %} NER label set {% endfile %} ### Color-coded labels We provide twelve default colors that you can configure during project creation or in the [Labels](/advanced/extensions/labels) extension. ![Label color palette](/files/J9SmyVL7TmDrSbiha79i) You can also create a `.csv` label set file with your desired label colors using HTML color codes. In the sample below, `label,color` is the header, indicating two columns. This will always be the first row in the `.csv` file. {% file src="/files/-MbjYI9KlLNdwRymQM3a" %} Colored label set {% endfile %} ``` label,color Annabeth Chase,#df3920 Harry Potter,#ff8000 Hermione Granger,#4db34d John Watson,#3399cc Percy Jackson,#cc3399 Sherlock Holmes,#9933cc ``` {% hint style="info" %} Colored label sets are only supported in the `.csv` format. {% endhint %} ### Limit selection to bottom-level labels only In hierarchical label sets, some labels act as broad categories while others are more specific. This setting restricts selection to bottom-level labels—those without child labels—to improve precision and consistency. #### When to use Use this setting when your project requires detailed and specific classification. Some examples: * **Fruit** * Apple * Banana * **Vegetable** * Carrot * Spinach When enabled, labelers can select **Apple, Banana, Carrot, and Spinach**, but not **Fruit** or **Vegetable**. #### How to configure You can enable this setting during project creation, in an existing project, or in the Label Management page. **During project creation** 1. Create new project. 2. Go to step 3 and select **Span labeling**. 3. In the **Label set** section, click the triple-dot menu on the label set and enable the setting. ![](/files/RTs53N2OhlhxXryBlYnm) **Within a project** 1. Open the **Labels** extension and select a label set. 2. Click the triple-dot menu and choose one of the following: 1. Add new label set 2. Replace existing label set 3. Edit label set 3. Expand the **Label set settings** accordion and enable the setting. ![](/files/ytMRquC1XTtWUoNKc0cI) **Label management page** 1. Go to the **Label management** page. 2. Select **Add label set** or click **Edit** icon on an existing label set. 3. Expand the **Label set settings** accordion and enable the setting. ![](/files/ozq2GcBRQ2vPSAphrwaD) #### Labeling behavior When this setting is enabled: * Only bottom-level labels can be selected. * Parent labels with child labels are not selectable. * Keyboard shortcuts (numbers, arrow keys, and `Enter`) apply only to bottom-level labels. ## Bounding box labeling ### **Label sets** You can use `.csv`, `.tsv`, or `.json` formats for bounding box label sets. * For `.csv` and `.tsv` files * We support **color names** (example: `red`), **hex values** (example: `#00FF00`), or **RGB** (example: `rgb(0,0,255)`). * You can also create a label set with only label names, as shown in the `Datasaur sample - Bbox only name.csv` file below. * Other properties, such as `captionAllowed` and `captionRequired`, will use default values if not specified. * For `.json` files * We support **hex values** and **RGB** formats only. {% file src="/files/iG5wYdtJYSKSIkIjYr1E" %} {% file src="/files/88e4WXaWyWBV7b9u2NRd" %} {% file src="/files/TXH3IJwCMJuq4vhBKCiH" %} {% file src="/files/IDatnggwXcViBmOVnXxs" %} {% file src="/files/idpJJxOEejnkoMFE4BO3" %} ### **Text caption** The **Allow text caption** setting allows labelers to add a caption to a bounding box. If disabled, captions cannot be added.
You can require text for specific labels by enabling the **Require caption** option.
## Row/Document labeling For row labeling and document labeling projects, a question set is a `.csv` file with questions in the first column and answers in the subsequent columns. | Column 1 | Column 2 | Column 3 | Column 4. | | | | ---------- | -------- | -------- | --------- | -------- | -------- | | Question 1 | Answer 1 | Answer 2 | Answer 3 | | | | Question 2 | Answer 1 | Answer 2 | | | | | Question 3 | Answer 1 | Answer 2 | Answer 3 | Answer 4 | Answer 5 | {% file src="/files/C8Z593J93QqKwdoezthI" %} Question set that only contain dropdown question type {% endfile %} You can also use a `.json` file to create a question set with multiple question types. {% file src="/files/i1PGcOaGbvbnkBPUo58w" %} Complex question set {% endfile %} ### Question hint ![](/files/LmckE0pJB1bzZY7PfHUx) You can optionally add a hint to each question. Hints can include instructions or explanations to help labelers provide more accurate answers. Each hint can contain up to 65,000 characters. You can configure question hints during project creation or in the [**Label management**](/workspace-management/label-management/question-set-management) page. ![Question hint in project creation](/files/8dvJRvryqMvzahkJ4ASd) ![Question hint in the label management page](/files/s0zIZkPB6VgvxCF1ySAB) #### Supported format The [Document labeling and Row labeling](/advanced/extensions/document-and-row-labeling) extensions support Markdown syntax in question hints. his allows you to format text, create lists, add links, or emphasize content using standard Markdown. | Formatting | Syntax | | ---------------------------- | ------------------------------------ | | **Bold** | **your text** | | *Italic* | *your text* | | Underline | \your text\ | | • Bullet | dashes (-) or asterisks (\*) | | 1. Numbering | 1., 2., 3., etc. | | [Link](https://example.com/) | \[your text] () | {% hint style="info" %} Markdown syntax counts toward the character limit (65,000 characters). {% endhint %} #### Best practices Keep hints brief and focused on relevant information. Long hints may appear as large text blocks and clutter the UI. For more complex content, consider including links that labelers can open when needed. ![Hint with markdown syntax](/files/6hmfVUiPPtpX6pTOOoO4) ![Rendered markdown in the extension](/files/ZCQY4JDXRGnM84eHOxir) ### Question types Here are the available question types for question sets. #### **1.** Text field **Text field** allows labelers to answer a question by entering a single line of free-form text.
You can add validation in the **More settings** accordion.
#### **2. Text area** **Text area** allows labelers to answer a question by entering multi-line free-form text.
#### **3. Dropdown** **Dropdown** requires labelers to answer a question by selecting one answer from a list.
You can upload a `.csv` file as an **answer set** for the dropdown options.
{% file src="/files/-MbjYI9TJbaP-3rI3X5N" %} You can also allow multiple selections by enabling **Allow multiple answers**.
#### **4. Hierarchical dropdown** **Hierarchical dropdown** allows labelers to answer a question by selecting from structures, multi-level options.
You can upload a hierarchical answer set after creating the question with [this format](#hierarchical-label-sets-or-dropdown-options). #### 5. True/False {% hint style="info" %} Previously called **Yes/No.** {% endhint %} **True/False** allows labelers to answer a question by checking the checkbox. You can also add a hint.
#### 6. Single choice {% hint style="info" %} Previously called **Radio button**. {% endhint %} **Single choice** allows labelers to answer a question by selecting one answer from up to 25 options.
You can also add a hint.
#### 7. Multiple choice **Multiple choice** allows labelers to answer a question by selecting several answers from up to 25 options. ![](/files/joZvTcDy3thbr1EnW5L3) #### **8. Date** **Date** allows labelers to answer a question by selecting a date from a calendar, ensuring valid date input. Labelers can still type it manually if needed.
You can set the current date as the default value by enabling the **Use current date as default value** setting in step 3 of project creation.
#### **9. Time** **Time** allows labelers to answer a question by selecting a time from a clock, ensuring valid time input. Labelers can still type it manually if needed.
You can set the current time as the default value by enabling the **Use current time as default value** setting in step 3 of project creation.
#### **10. Slider** **Slider** allows labelers to answer a question by selecting a numeric value within a defined range (example: from 1 to 10).
You can hide the value from labelers if needed (it remains visible in reviewer mode).
You can customize the slider color using predefined options or custom values ([hex, color name, or RGB](https://developer.mozilla.org/en-US/docs/Web/CSS/named-color)).
You can also use the interactive preview to test the slider.
When you change the **minimum** or **maximum** value of a slider question, existing answers outside the new range will trigger an error message in the **Row labeling** or **Document labeling** extension when users select the affected row or document. {% file src="/files/AQd4DVNYsYUFS5YKi3wT" %} #### **11. URL** **URL** allows labelers to input links with validation.
#### 12. Grouped attributes **Grouped attributes** allows multiple related questions to be grouped together.
#### **13. Script-generated questions** {% hint style="info" %} Available only for **Row labeling** projects. {% endhint %} **Script-generated questions** dynamically generates questions for each row based on its data, allowing more flexible and dynamic workflows. For more details, see this page [here](/advanced/script-generated-question).
### **Advanced settings** In row labeling projects, you can use the **Refer answer to table column** option in **Advanced settings** to pre-fill answers based on your dataset. See the [full guide](/data-studio-projects/lets-get-labeling/label-sets/refer-answer-to-table-column) for more details. ### **Answer validation script** {% hint style="info" %} This feature is available only in **row labeling** projects and is disabled by default. Please reach out to [**support@datasaur.ai**](mailto:support@datasaur.ai) if your team needs this feature. {% endhint %} The a**nswer validation script** is a TypeScript-based feature that validates answers in row labeling tasks. It allows you to define custom validation logic for complex scenarios, such as validating answers using other questions, comparing data across questions, or calling external APIs. If a submission fails validation, an error message is shown. This feature improves control, accuracy, and consistency in the labeling process. ![](/files/pw2ACAdFP46OkyWp0FoM) #### **Key capabilities** 1. **Row-specific validation**: Validates data based on the current row. 2. **Cross-question validation**: Compares answers across multiple questions. 3. **API-based validation**: Use external APIs or external business logic for validation. #### Configure the validation script {% hint style="info" %} Validation can only be configured after questions are set up. Only admins can access this feature in reviewer mode. {% endhint %} 1. Go to the **Row labeling** extension in a project. 2. Click on the three-dot menu. 3. Select **Configure answer validation script**. ![](/files/3i3q7zrXa0lS6z2zPgCF) ![](/files/FUWifYLdJa0nQYdueAeW) When you open the **Configure answer validation script** dialog for the first time, a template is provided.
View Template ```tsx /** * This function should return a ValidationResult object. * * @param {ValidationArgs} args - The arguments for the validation. * @returns {ValidationResult} The result of the validation. * @example * To return an error: * return { errorMessage: "This is a sample error message." }; * To return no error: * return {} */ async ({ columns, row, questions, answers }: ValidationArgs): Promise => { **// TODO: Implement your validation logic here.** return {}; /** Helper functions with access to the validation args */ /** * Get the answer for a question by its label. * For grouped attributes, pass in the questions array from the grouped attribute. * @param label - The label of the question. * @param searchQuestions - Optional. The questions to search through. Defaults to the questions in the validation args. * @returns The answer for the question. */ function getAnswerByQuestionLabel(label: string, searchQuestions: Question[] = questions) { const question = searchQuestions.find((q) => q.label === label); if (!question) throw new Error(`Could not find question with label: "${label}".`); return answers[`Q${question.id}`]; } /** * Get the value of a cell by its column label. * @param label - The label of the column. * @returns The value of the cell. */ function getCellValueByColumnLabel(label: string) { const column = columns.find((column) => column.name === label); if (!column) { throw new Error(`Couldn't find column with label: "${label}".`); } const cell = row.find((row) => row.index === column.id); if (!cell) { throw new Error(`Couldn't find cell with index: ${column.id}. Column: "${column.name}"`); } return cell.content; } }; ```
To provide feedback when a submission fails, return an object with an `errorMessage` property (optional). When the submission passes, return an empty object. {% tabs %} {% tab title="🔴 Failing the submission" %} ```tsx async ({ columns, row, questions, answers }: ValidationArgs): Promise => { // this script will always prevent the labeler to submit the answer. return { errorMessage: 'Please double-check your answers.' }; } ``` {% endtab %} {% tab title="🟢 Passing the submission" %} ```tsx async ({ columns, row, questions, answers }: ValidationArgs): Promise => { // this script will always allow the labelers to submit the answer. return {}; } ``` {% endtab %} {% endtabs %} When validating, you may need to access certain information to determine whether the answer is valid or needs adjustment before submission. You can access all required data from the function argument, as shown in the example function. * `columns: TableColumn[]` contains information about the column structure of the data being labeled.
View structure ```tsx interface TableColumn { id: number; name: string; displayed: boolean; labelerRestricted: boolean; rowQuestionId?: number; } ```
* `row: Cell[]` is an array of cells containing the data that is being labeled.
View structure 
interface Cell {
  line: number;
  index: number;
  content: string;
  tokens: string[];
  metadata?: CellMetadata[];
}

interface CellMetadata {
  key: string;
  value: string;
  type?: string;
  pinned?: boolean;
  config?: TextMetadataConfig;
}

interface TextMetadataConfig {
  backgroundColor: string;
  color: string;
  borderColor: string;
}
* `questions: Question[]` is an array of questions in the project.
View structure ```tsx enum QuestionType { DROPDOWN = 'DROPDOWN', HIERARCHICAL_DROPDOWN = 'HIERARCHICAL_DROPDOWN', NESTED = 'NESTED', TEXT = 'TEXT', SLIDER = 'SLIDER', DATA = 'DATA', DATE = 'DATE', TIME = 'TIME', CHECKBOX = 'CHECKBOX', URL = 'URL', } enum SliderTheme { PLAIN = 'PLAIN', GRADIENT = 'GRADIENT', } interface Question { id: number; label: string; type: QuestionType; required: boolean; activationConditionLogic?: string; bindToColumn?: string; config: QuestionConfig; } interface QuestionConfig { multiple?: boolean; // TEXT maxLength?: number; minLength?: number; pattern?: string; multiline?: boolean; // SLIDER theme?: SliderTheme; min?: number; max?: number; step?: number; // DATE TIME format: string; defaultValue?: string; // DROPDOWN HIERARCHICAL options?: Array<{ id: string; label: string; parentId?: string | null }>; // GROUPED questions?: Array; // CHECKBOX hint?: string; } ```
* `answers` is an object where each key is a question ID and each value is its corresponding answer. Depending on the question type, the value can take one of these four formats: | | multiple: false | multiple: true | | ------------------ | --------------- | -------------- | | normal question | string | string\[] | | grouped attributes | Answer | Answer\[] |
View structure 
type Answer = string | string[] | Answers[];

interface Answers {
  [questionId: string]: Answer;
}
We provide helper functions in the template to simplify common data access tasks, such as: * `function getCellValueByColumnLabel(label: string): string;` This function returns the value of a cell based on the column label. * `function getAnswerByQuestionLabel(label: string, searchQuestions: Question[] = questions): Answers;` This function returns the answer based on the question label. #### Validating answer through an API call You can include API requests in your validation logic using the [Fetch API](https://developer.mozilla.org/en-US/docs/Web/API/Fetch_API), enabling dynamic or third-party validations. {% hint style="warning" %} **Disclaimer** * We are not responsible for API calls that are misrouted, misconfigured, or sent to unintended parties, which may result in data exposure or compromise. * You are responsible for ensuring the accuracy, security, and integrity of your API configurations and transmissions. By using this feature, you acknowledge and accept these responsibilities. {% endhint %} #### Examples
Validating answers between two questions ```tsx async ({ questions, answers }: ValidationArgs): Promise => { const questionAAnswer = getAnswerByQuestionLabel("Question A"); const questionBAnswer = getAnswerByQuestionLabel("Question B"); if (questionAAnswer !== questionBAnswer) { return { errorMessage: "The answer to Question A must match Question B." }; } return {}; // Pass the validation // ...existing helpers provided by template }; ```
Validating an answer based on a cell value ```tsx async ({ columns, row, questions, answers }: ValidationArgs): Promise => { const statusColumnValue = getCellValueByColumnLabel("Status"); const approvalStatusAnswer = getAnswerByQuestionLabel("Approval Status"); if (statusColumnValue === "Complete" && approvalStatusAnswer !== "Approved") { return { errorMessage: "If 'Status' is 'Complete', 'Approval Status' must be 'Approved'." }; } return {}; // ...existing helpers provided by template }; ```
Validating an answer through an API request using the Fetch API ```tsx async ({ columns, row, questions, answers }: ValidationArgs): Promise => { const response = await fetch('', ...); const data = await response.json(); // assume the API returns a JSON object: { result: 'valid|invalid', message: 'the error message' }; if (data.result === 'invalid') { return { errorMessage: data.message }; } return {}; // ...existing helpers provided by template } ```
#### **FAQs** 1. **Can I validate across multiple rows?** No, the validation script is row-specific and operates on each row individually. 2. **What happens if there is an error in the script?** Unhandled errors or exceptions will trigger a validation error and prevent submission. You can handle errors within the script to allow submission if needed. ## **Hierarchical label sets or dropdown options** You can upload multi-level hierarchical label sets for span labeling projects, and hierarchical dropdown options for row labeling or document labeling projects. The following example shows a supported `.csv` format: {% file src="/files/CGftTJyXD6E7jfKt1aZL" %} ``` id,label 1,Novel 1.1,Author 1.1.1,Name 1.1.2,Works 1.2,Title 1.2.1,Main Title 1.2.2,Subtitle 2,Characters 2.1,Antagonist 2.2,Protagonist ``` ### **File structure** #### **1. Header** The header **`id,label`** will always be the first row in the `.csv` file. The first label or option should have `1` as the ID, just like in the example above. #### **2. ID format** IDs follow a hierarchical numbering format similar to Microsoft Word: * `Novel` is the root level with ID `1`**.** * `Author` is a second-level category under `Novel` with ID `1.1`. * `Name` is a third-level item under `Author` with ID `1.1.1` {% hint style="info" %} **Important notes** When importing data, the CSV format uses dots (`.`) to represent hierarchical relationships. However, these dots are automatically converted into a different ID structure in the JSON format because dots are reserved for path traversal operations in the system. **This means dots must not be used in JSON IDs**. Here's how it works: * CSV Input: ```typescript id,label 1,Novel 1.1,Author ``` * Will be converted to JSON as: ```json [  {    "label": "Novel",    "id": "1"  },  {    "label": "Author",    "id": "2",    "parentId": "1"  } ] ``` In JSON format: * ✅ Correct: `"id": "2"` * ❌ Incorrect: `"id": "1.1"` Using dots for JSON IDs will cause incorrect path resolution when selecting items in the hierarchy. {% endhint %} #### 3. Hierarchical label set in span labeling projects The hierarchy will be visible in the **Labels** extension and the label box. You can also use the same label name under different parent labels. ``` id,label 1,Software 1.1,Java 2,Geography 2.1,Java ``` In the example above, although `Java` appears twice, each instance belongs to a different parent, making it contextually unique.
Using the same label name more than once under the same parent is not allowed. In the example below, the system flags an error because both `Apple` entries are under `Fruit`. ``` id,label 1,Fruit 1.1,Apple 1.2,Apple ``` #### 4. Hierarchical dropdown options in row or document labeling projects Choose **Hierarchical dropdown** as the question type when creating the project. The hierarchy will be displayed in the **Row labeling** and **Document labeling** extension and the answer column in the table. ![](/files/JLOPxARxmKI0rdTuhaPH) {% hint style="info" %} In the dropdown menu: * Clicking the **Home** icon navigates to the top-level label. * You can search for **bottom-level options** globally. {% endhint %} # Refer Answer to Table Column In row labeling projects, you can speed up labeling by pre-filling answers based on data in your uploaded file (`.csv`, `.xlsx`, etc.). To do this, configure **Refer answer to table column** for each question and map it to a specific column header in your dataset. The system then automatically populates answers using values from that column. To configure this during project creation, go to step 3. Add a question, then open the **More settings** accordion, then select the column header the question should reference.
After completing the setup, open the project. The question is pre-filled with answers from the referenced column.
## **Configuring for nested questions** Datasaur also supports **Refer answer to table column** for nested questions (child questions under a [grouped attributes](https://docs.datasaur.ai/data-studio-projects/lets-get-labeling/label-sets/pages/sjMjHjQsHtR5BrPf3PUF#id-12.-grouped-attributes) question). The grouped attributes question cannot reference a column directly, so you must configure the reference on its child questions. You can configure this using the same steps as for regular questions. Parsing behavior may vary depending on whether the parent or child questions allow multiple answers, as nested questions may need to split values into multiple groups, multiple items, or both. The next section explains how parsing works. ## Data parsing and separators To support pre-labeling, certain characters in your data are used to split and group answers. ### **Multiple answers separator (**`;`**)** If a question **allows multiple answers**, use a semicolon `;` to split values in a single cell into separate answers. If the question **does not** allow multiple answers, the semicolon is ignored, and the entire cell value will be treated as a single pre-labeled answer. **Example** * **Question:** `Fruit?` (Type: Text) * **Referred column cell data:** `Apple;Banana;Pear` * **Result:** The system creates three pre-labeled answers: **Apple**, **Banana**, and **Pear**. {% file src="/files/8Xc9p3jIYvt7n5Ww63xV" %}
### **Answer group separator (**`|||`**)** {% hint style="info" %} This separator is used only for **nested questions in grouped attributes** question. {% endhint %} The `|||` separator splits values in a single cell into **multiple answer groups**. This separator only works when the parent grouped attributes question **allows multiple answers**. Otherwise, all `|||` separators are ignored, and the cell is treated as a single group. You can also use `|||` together with the multiple answers separator (`;`) to split values within each group. **Requirements for group parsing** * The **grouped attributes (parent)** question must be configured to **allow multiple answers**. * The **nested question (child)** must reference the column containing the separators. * The child question can optionally **allow multiple answers** to split values using `;` within each group The system parses values differently based on your configuration: * **Case 1**
* **Parent question:** Allow multiple answers → Unchecked :x: * **Nested question:** Allow multiple answers → Checked :white\_check\_mark: * **Column data:** `A1;A2|||B1;B2` * **The resulting pre-labeled answer:** * Group 1: `A1`, `A2|||B1`, `B2` * **Case 2**
* **Parent question:** Allow multiple answers → Checked :white\_check\_mark: * **Nested question:** Allow multiple answers → Unchecked :x: * **Column data:** `A1;A2|||B1;B2` * **The resulting pre-labeled answer:** * Group 1: `A1;A2` * Group 2: `B1;B2` * Case 3
* **Parent question:** Allow multiple answers → Checked :white\_check\_mark: * **Nested question:** Allow multiple answers → Checked :white\_check\_mark: * **Column data:** `A1;A2|||B1;B2` * **The resulting pre-labeled answer:** * Group 1: `A1, A2`\ Group 2: `B1, B2` ## Test data example 1. Upload the file below during project creation: {% file src="/files/2nbjDqbUiILVYI41no8x" %} 2. In step 2, enable **Convert first row as header**. ![](/files/OXTDC72BeGHpz5DPP8AK) 3. In step 3, configure the questions as follows: * **Question 1:** `Colors` (Type: Text) * Enable **Allow multiple answers** * Set **Refer to column:** `colors_available` ![](/files/wptpFDUJQttGBBWwlbNU) * **Question 2:** `Vehicle parts` (Type: Grouped attributes) * Enable **Allow multiple answers** * **Question 2.1:** `Part name` (Type: Text) * Enable **Allow multiple answers** * Set **Refer to column:** `part_names_and_tags` ![](/files/eyio2ioAfyumD5was8Pv) ### **Result** After the project is created, the questions are automatically pre-filled based on the values in the `.csv` file. ![](/files/KXghWEUM2yHB09OIWs63) # Dynamic Question Set A dynamic question set lets you show or hide questions based on answers to other questions. You can set up this logic when [creating a project](/data-studio-projects/creating-a-project#step-3-labeler-tasks-sharpvartagopen-if-renderprojectcreationtutorialvideolink-vartagclose-video-tut)[ ](/data-studio-projects/creating-a-project#step-3-labeler-tasks-sharpvartagopen-if-renderprojectcreationtutorialvideolink-vartagclose-video-tut)or from the [Question set management](/workspace-management/label-management/question-set-management) page. To set it up: 1. Click **Manage question logic** icon located at the bottom right of the question list area.
2. Define the logic. 1. **Always show:** The question is always shown and does not depend on other answers. 2. **If:** The question is shown only when specific conditions are met.
## Rule details ### **Comparison rule**
Each question type supports specific comparison rules, which let you define conditions based on the answer type. #### Common rules * `is`: Checks if the answer exactly matches a value. * `is answered`: Checks whether the question has been answered. * `contains`: Checks whether the answer includes a specific keyword. * **Single answer**: Checks if the keyword is part of the answer (Example: `run` is a substring of `running`). * **Multiple answers**: Checks for an exact match (Example: `run` exactly matches `run`). * `starts with / ends with` : Checks whether the answer begins or ends with specific text. * `is checked / is unchecked`: Checks whether a checkbox is selected or not. **Examples:** * Text question * **Q1**: A text question with a single answer * **Q2**: A text question with logic applied: `Q1 CONTAINS run`. * If you answer **Q1** with `Marathon run` or `running outside`, **Q2** will be activated. * Dropdown question * **Q1**: A dropdown question with a single answer * **Q2**: A text question with logic applied: `Q1 CONTAINS run.` * If you select `Marathon run` or `running outside` in **Q1**, **Q2** will be activated. #### Numeric rules * `> / <`: Checks if a value is greater or less than a specific number. * Example: * **Q1**: A slider question with a single answer. * **Q2**: A text question with logic applied: `Q1 < 8`. * If you set **Q1** to `7` or `6`, **Q2** will be activated. * `≥ / ≤`: Checks if a value is greater than or equal to, or less than or equal to a specific number. * Example: * **Q1**: A slider question with a single answer. * **Q2**: A text question with logic applied: `Q1 ≥ 8`. * If you set **Q1** to `8`, `9`, or `10`, **Q2** will be activated. To understand which rules apply to each question type, refer to the table below.
Question type/ruleisis answeredcontainsstarts withends withis checkedis unchecked>≥<≤
Texttruetruetruetruetruefalsefalsefalsefalsefalsefalse
Text areatruetruetruetruetruefalsefalsefalsefalsefalsefalse
Dropdowntruetruetruefalsefalsefalsefalsefalsefalsefalsefalse
Hierarchical dropdowntruetruetruefalsefalsefalsefalsefalsefalsefalsefalse
Datetruetruefalsefalsefalsefalsefalsefalsefalsefalsefalse
Timetruetruefalsefalsefalsefalsefalsefalsefalsefalsefalse
Checkboxfalsefalsefalsefalsefalsetruetruefalsefalsefalsefalse
Radio buttontruetruefalsefalsefalsefalsefalsefalsefalsefalsefalse
Slidertruefalsefalsefalsefalsefalsefalsetruetruetruetrue
URLtruetruetruetruetruefalsefalsefalsefalsefalsefalse
### **Conjunction rule** For conditional logic, you can use two conjunctions: * `AND`: All conditions must be true for the question to appear. * `OR`: Only one condition needs to be true for the question to appear. # Multiple Label Sets A single project may require multiple labeling focuses. Datasaur supports this by allowing multiple label sets within the same project. Multiple label sets lets you apply different labeling focuses to the same data. For more details about label sets, see the [documentation](/data-studio-projects/lets-get-labeling/label-sets#span-based-labeling). You can create multiple label sets in the same way as a single label set—either manually or by uploading a label set file during project creation or from the **Labels** extension. ### Add multiple label sets during project creation In step 3, Datasaur automatically creates `Untitled Label Set 1`.\ You can rename, add, remove, and reorder label sets as needed. You can also drag labels to the right to nest them under other labels. To add a new label set, click the **Add new label set** button in the label set list. A project can have up to 10 label sets.

Select the [ + ] icon to add an additional label set

### Add multiple label sets from Labels extension After the project is created, you can still add, edit, or remove label sets from the **Labels** extension. To add a new label set, click the three-dot menu and select **Add new label set.**
You can create the label set manually or upload the label set file there.
{% hint style="warning" %} Deleting a label set removes all applied labels from that label set in the project. {% endhint %} ## Active label set The active label set is the one currently selected for use in the project. You can change it from the dropdown above the labeling area. For example, if you have both `NER` and `Stanford` label sets, you can switch between them by selecting the desired label set.
After switching label sets, you can start labeling using the active label set.
## Multiple label sets in a span labeling project with arrows In span labeling project, you can draw relational arrows between labeled spans. Some projects require labeling both spans and the relationships between them. To enable this, turn on **Allow arrows to be drawn between labels** in step 3 during project creation or from the project settings. You can then add both span label sets and arrow label sets. If you do not need to label arrows, an arrow label set is not required.
After the project is created, the interface shows a separate label set dropdown (with an arrow icon) for relationship labels.
Click the dropdown and select the appropriate label set.
You can now draw arrows and apply the corresponding labels.
## Export Projects with multiple label sets can be exported to [.json\_advanced](/compatibility-and-updates/supported-formats#json_advanced) and [.tsv\_non\_iob](/compatibility-and-updates/supported-formats#tsv_non_iob). # Reviewing Projects ## Overview Reviewer mode helps you review labeling work and ensure data quality. As a reviewer, you check accuracy, resolve conflicts, and keep the workflow consistent. ## How it works? You must have the **Reviewer** role to use the reviewer mode in a project. [Learn more about roles here](/workspace-management/role-and-permission). ### Reviewer mode for span labeling In span labeling projects, there are three types of conflicts: * Content conflict * Span conflict * Arrow conflict See the [documentation](/advanced/extensions/review#conflicts) for details on the differences between these conflict types. #### Resolve conflicts To resolve a conflict, right-click the conflicted label to accept or reject it. You can also replace it by left-clicking the label and selecting a new one.
#### Navigate between conflicts You can navigate between conflicts from the **Go** menu or by pressing `Alt` + `Shift` + `Right` to go to the next conflict and `Alt` + `Shift` + `Left` to go to the previous one.
#### Label status colors In reviewer mode, label colors indicate the status of each label. You can view the color legend in the status bar at the bottom.
* **Gray** labels mean they have reached consensus among labelers.
* **Yellow** labels indicate that they were created using the [asisted labeling extensions](/assisted-labeling/ml-assisted-labeling).
* **Blue** labels mean the label has been rejected by the reviewer. By default, rejected labels are hidden. You can make them visible in **File** → **Settings** → **Administration** by enabling **Show rejected labels in reviewer mode**.
* **Purple** labels indicate that they were applied by the reviewer.
* **Red** labels indicate that they are conflicted (have not reached the consensus threshold) and require review to be accepted or rejected.
In span labeling, you can see the number of applied labels, the last labeled row, and the total resolved rows. This information is shown in the bottom-right corner.
### Reviewer mode for row labeling Reviewing in row labeling focuses on accepting, rejecting, or updating answers in each row using the **Row labeling** extension.
#### Row colors Each row has a colored line on the left side of the number column to indicate its status.
* **No colored line** means the answers for that row have reached consensus among labelers or have already been resolved by the reviewer. * **A red line** indicates that the row has no consensus. All answers are conflicted. * **A pink line** indicates that the row contains both a consensus answer and a conflicting answer. This occurs when a question allows multiple answers.
#### Answers in the table Whether answers are shown in the table depends on their status. * **Meets consensus:** Answers are displayed. * **Conflicted (No consensus):** No answers are displayed. * **Mixed consensus and conflict:** Only consensus answers are shown; conflicted answers appear after review. * **Resolved by reviewer:** Final answers are displayed after review. Submitting answers in the **Row labeling** extension updates the table accordingly. ### Reviewer mode for document labeling The behavior is similar to row labeling, with the main interactions happening in the **Document labeling** extension.
### Reviewer mode for bounding box labeling Review for bounding box labeling is not yet available. This feature is coming soon!
# Review Sampling Review sampling helps reviewers track progress and focus on a subset of data. This allows for efficient quality checks while reducing review time. {% hint style="info" %} This feature is only available for span labeling, row labeling, document labeling, and span + document labeling projects. {% endhint %} ### Configuration You can enable this feature during project creation or in project settings. #### During project creation 1. In step 4, set the conflict resolution to **No consensus**. 2. Enable **Set review sampling rate** in the bottom right corner. 3. The default sampling rate is 10%, but you can adjust it as needed. ![](/files/fncgMrmMEKuhPOZRbPRx) #### **In project settings** * In project settings, go to the **Assignment** tab. * Set the conflict resolution to **No consensus**. * Enable **Set review sampling rate** in the bottom right corner.
### View sampling progress Once sampling rate is enabled and labeling is complete, reviewers can track progress in the **Progress** tab of the **Review** extension. This shows how many samples need review and how many have been completed.
### How the progress is calculated #### Span labeling projects Progress is based on the number of unique labeled spans in each document. **Example**: If Document A has 8 labeled spans and Document B has 11 labeled spans, with a 50% sampling rate, the reviewer needs to review 4 spans from Document A and 6 spans from Document B. #### Row labeling projects Progress is based on the number of rows in each document. **Example**: If Document A has 10 rows and Document B has 5 rows, with a 50% sampling rate, the reviewer needs to review 5 rows from Document A and 3 rows from Document B. #### Document labeling projects Progress is based on the total number of documents. **Example**: If a project has 12 documents and a 50% sampling rate, the reviewer needs to review 6 documents. #### Span + document labeling projects Progress combines both span and document calculations. **Example**: If a project has 12 documents, each with 8 labeled spans, and a 50% sampling rate, the reviewer needs to review 6 documents and at least 4 spans within each document. ### Marking the project as complete When the feature is enabled, the system checks whether the required sampling rate is met before allowing the project to be marked as complete. * If the requirement is met, a confirmation dialog appears and you can mark the project as complete. ![](/files/gA44JNl4lXkIaHgsTZRx) * If the requirement is not met, a warning shows how many items remain. You must complete the required reviews before marking the project as complete. ![](/files/ol2BmWslyqdKmn9vlSK1) This ensures the review process meets the required quality threshold before completion. # Export Project ## Export projects Both features are supported through API call. Click [here](/api/export-project) for more detailed explanation. 1. Go to the **File** menu. 2. Select **Export file** or **Export all files**.
* **Export** **file:** Export only the currently opened file. The output reflects the latest project state and does not include all project data. * **Export all files:** Export all files in the project. For projects with multiple assignees, each labeled version is exported separately. The output is a `.zip` file containing three folders: * **DOCUMENT-Labeler-name:** Contains files labeled by each labeler. * **REVIEW:** Contains the final version, including auto-accepted labels and reviewer edits. * **ROOT:** Contains the original raw text without any labels or edits. ## Advanced settings When exporting, you can configure additional settings to control what gets included in the export.
### Export work from assignee {% hint style="info" %} This setting is only visible to Supervisor and Admin roles, and is only supported when using export all files in the Reviewer mode. {% endhint %} Choose whose work to include in the export: * **All labelers** — Includes work from every labeler. * **All reviewers** — Includes work from every reviewer. * **Specific labeler** — Includes work from one or more selected labelers. ### **Fields to export** {% hint style="info" %} Fields to export is only available for the **Datasaur Schema** and **File Transformer**, and is currently only supported for **Span+Line** and **Row labeling** projects. {% endhint %} Select which fields to include in the export output: * **Line numbers** — Identifier for each row. * **Contents** — Original text content from the documents. * **Span labels** — Label classes applied to spans. * **Row/Line answers** — Question-answer pairs applied to individual rows/lines. Available only in row/line labeling projects. * **Metadata** — Additional data fields associated with the content. * **Comments** — Comments added to labels, relationships, or spans. * **Label sets** — Collections of label classes used for labeling. * **Row/Line questions** — Collections of questions used for row/line answering. In File Transformers, these fields are available as secondary, nullable inputs. You can customize your export transformer based on the selected fields. See the attached file for an example File Transformer that outputs the document together with the export configuration. Feel free to try it out and modify it to suit your needs. {% file src="/files/tpmshKCeNiJnTGe3wD4o" %}
### Include unresolved labels / answers {% hint style="info" %} This is supported for **span**, **row**, and **document labeling** projects in both **Datasaur Schema** and **CSV** formats. {% endhint %} You can include unresolved labels or answers inside the export result. #### Enable the option When you select a supported export format, a checkbox will appear. If selected, unresolved labels or answers will be included in the export. #### Export result * **Comma-separated values (.csv)** * **New column: Label Status** This column indicates whether a line is conflicted or resolved. * **New column: Line** This column indicates the line number. * **Datasaur Schema (.json)** * **New values:** `rowAnswers`, `documentAnswers`, `spanLabels`, `arrowLabels` The conflicted values will be added to `rowAnswers`, `documentAnswers`, `spanLabels`, or `arrowLabels`. You can differentiate between resolved and unresolved answer by looking at `labeledBy` attribute. Unresolved label should have **CONFLICT** as their `labeledBy` value. The following section will give you some illustration on the result. * **Span labeling** * **Datasaur Schema (.json)** Conflicted labels will be added to `spanLabels` or `arrowLabels`. * **Comma-separated values (.csv)** This format is similar to the Amazon Comprehend CSV export format, but with an additional column titled `Label Status`. * **Row and document Labeling** * **Datasaur Schema (.json)** Unresolved answer will not be added to the answer set (`rowAnswerSets` for Row Labeling, while `documentAnswerSets` for Document Labeling) However, it will be added to `rowAnswers` for Row Labeling, while `documentAnswers` for Document Labeling, along with the resolved answers. * **Comma-separated values (.csv)** Adds `Label Status` and `Line` column. There may be cases where a single line contains both resolved and unresolved answers due to consensus. In such cases, the answers are split into two lines: one for resolved answers and one for unresolved answers. ## Export methods When exporting a file, you can choose from several delivery options. All methods are also available via [API](https://datasaurai.gitbook.io/datasaur/api/apis-docs). ![](/files/fwoAFhIr2lu45YPvVBPD) ### **Download** * The export result is uploaded to Datasaur's storage bucket, and you download it directly to your device via a generated link. * Keep in mind that the time needed to generate the link will be directly proportional to the size of the project. ### **Email** * Datasaur will generate a download link and send it to the email address used for your account. * The link **expires in 6 hours**. ### **Webhook** * The export result is sent as a payload in a webhook request. * See the [Webhook](/api/export-project/custom-webhook) documentation for full details. * The link **expires in 6 hours**. ### **External Object Storage** * The export result is uploaded directly to your selected external storage bucket. * You can also add a prefix to the file name. The prefix is appended directly without a separator. For example: * Prefix: `test` * File name: `name.json` * Result: `testname.json` ## Export multiple projects from the Projects page You can export multiple projects with the **same project settings** in the **Projects** page. 1. Select all projects you want to export.
2. Click **Export** button. You can choose the export format and method. The output will be a `.zip` file.
{% hint style="info" %} We recommend exporting up to 10 projects at once for better performance. {% endhint %} # Labeling Agent Labeling agents allow you to assign ML models as labelers in your project and evaluate their performance alongside human labelers. This helps you understand which labeling approach works best for your needs, whether human, machine, or both. Labeling agents simplify the process of testing and comparing ML models inside Datasaur: * You no longer need to create separate accounts or log in as the model to run predictions. * Model outputs are now part of the same analytics and comparison tools used for human labelers. * It’s easier to measure performance and decide what labeling strategy to use. {% hint style="info" %} Models must be deployed in [**LLM Labs**](/llm-projects/deployment) in the same team workspace as the Data Studio project. {% endhint %} ## Create a labeling agent {% hint style="info" %} **Supported labeling types**: Span labeling, row labeling. {% endhint %} #### 1. Prepare the label set Make sure the label set you will be using matches the one you configure in step 2. Below is a simple example of labels that can later be used in Data Studio: {% code overflow="wrap" %} ```json { "name": "Labeling agent Label set", "options": [ { "id": "NhsjWIgaAQH3g6dsvtW6a", "color": "#f93b90", "parentId": null, "label": "PERSON" }, { "id": "X1bKK7Nxf9SGaBfDpzH7g", "color": "#d4e455", "parentId": null, "label": "DATE" }, { "id": "NP2RJr7tD5aMfVBnG6TOm", "color": "#85c98e", "parentId": null, "label": "ORG" } ] } ``` {% endcode %} #### 2. Define your instructions In LLM Labs, create a new sandbox and set up the model to act as a labeling agent. To help the model understand what to label, you’ll need to provide clear system and user instructions. The output of the model must be in JSON object format, aligned with the label set defined in your NLP project, to ensure compatibility with regex-based string matching for labeling in your NLP platform. Below is an example setup: **System instruction** ``` You are an expert data labeler ``` **User instruction** {% code overflow="wrap" %} ```` Given the document text, please extract the following information and present it in JSON format as shown below: PERSON: People, including fictional. DATE: Absolute or relative dates or periods. ORG: Companies, agencies, institutions, etc. Instructions Summary: 1. Extract and present the information in the specified JSON format. 2. Ensure that all extracted data is accurate and corresponds directly to the content of each document. Return the value of extracted fields in JSON structure in plain text, following this JSON FORMAT { "PERSON": ["People, including fictional."], "DATE": ["Absolute or relative dates or periods."], "ORG": ["Companies, agencies, institutions, etc."], } VERY IMPORTANT RETURN THE ANSWER WITHOUT ```json ANSWER PRECISELY GIVEN FROM THE SENTENCE PROMPT AND DON'T MASK THE ANSWER, ANSWER BASED ON THE GIVEN SENTENCE ```` {% endcode %} #### 3. Test with a prompt example To check if your instructions work as expected, you can test them using an example sentence. Here's how you might write a prompt: {% code overflow="wrap" %} ``` Label set: - PERSON - DATE - ORG Sentence: Ivan Lee is the CEO and Founder of Datasaur.ai. He graduated with a Computer Science B.S. from Stanford University. He was chosen for the selective Mayfield Fellows entrepreneurship program in 2010. Ivan went on to found Loki Studios, an iOS game studio. After raising institutional funding from DCM's A-Fund and launching a profitable game, Loki was acquired by Yahoo. ``` {% endcode %} After you click the **Run** button, the expected output will be: {% code overflow="wrap" %} ``` { "PERSON": ["Ivan Lee"], "DATE": ["2010"], "ORG": ["Datasaur.ai", "Stanford University", "Mayfield Fellows", "Loki Studios", "DCM's A-Fund", "Yahoo"] } ``` {% endcode %} #### 4. Deploy the model You need to deploy the model first before it becomes available and visible in Data Studio as a labeling agent.
## Assign labeling agents #### [Span Labeling Agent](/agent/labeling-agent/span-labeling-agent) Use this guide if your project requires the agent to apply labels to spans of text. #### [Row Labeling Agent](/agent/labeling-agent/row-labeling-agent) Use this guide if your project requires the agent to answer row-based questions. ## **Launch the project and review results** When you click Launch project, the labeling agents will automatically start labeling. **Current limitation:** * Each span will only have one label. * Limited supported question type for optimized performance. ## Best practices * Use the external model as a timesaving aid but always include a human review step. * Train your model with high-quality data to improve suggestion accuracy. * Communicate clearly with labelers about how to handle model predictions. * Automate some of the work with consensus by using multiple models, e.g., use the consensus of 3 and deploy 3 Labeling Agents, then focus only on those that are not accepted through consensus. ## FAQs * **Can I assign multiple models to the same project?** * Yes. You can assign up to 10 labeling agents. * **Can I use labeling agents in line labeling project?** * Not yet. They can be assigned to span + line labeling project but will only apply labels for span labeling task. * **How are labeling agent labels shown in the text editor?** * They are treated like human labelers but are masked. You’ll see their labels in the reviewer mode and analytics. # Span Labeling Agent Use a span labeling agent when you want a model to automatically apply labels to [Span Based](https://docs.datasaur.ai/data-studio-projects/lets-get-labeling/span-based#adjust-span-selection) project type. This is useful for tasks such as named entity recognition, information extraction, and other text annotation workflows where the model needs to highlight and classify specific text spans. A span labeling agent works alongside human labelers, so you can compare model output, review quality, and measure performance in the same project. {% hint style="info" %} Models must be deployed in [**LLM Labs**](/llm-projects/deployment) in the same team workspace as the Data Studio project. {% endhint %} ### Before you begin Before assigning a span labeling agent, make sure you have: * a deployed model in LLM Labs * a span labeling project in Data Studio * a label set prepared for the project * prompt instructions that clearly define what spans the model should label ### How span labeling agents work For span labeling projects, the labeling agent reads the input text and returns predicted spans based on the label set you configure. The output is then applied in the project as model-generated annotations. A typical setup flow looks like this: {% hint style="info" %} Pre-requisite: [Deployed model in LLM Labs](https://docs.datasaur.ai/agent/labeling-agent) {% endhint %} 1. Assign the deployed model as a labeling agent in Data Studio 2. Launch the project and review the generated labels ### Assign the model as a span labeling agent To assign a deployed model as a labeling agent: 1. Go to **Projects** and click **Create new project**. 2. Upload your files and Select **Span Labeling**. 3. Configure Labeler’s tasks, you can also enable *Allow arrows to be drawn between labels* checkbox if needed. 4. In the **Assignment** step, open the **Labeling agents** tab. 5. Select the deployed models you want to assign as labeling agents.

Selecting deployed LLM Labs Sandbox as labeling agent

6. You need to set the agent task by clicking Set a default agent task or Set task scope on individual labeling agent, and complete the project setup. #### Configure span labeling tasks
You can configure: 1. **Label set**: the label set the agent should use. 2. **Allow the labeling agent to apply arrow**: available only if *Allow arrows to be drawn between labels* is enabled during project setup. **Allow the labeling agent to apply arrow:** this option is only available if you enable *Allow arrows to be drawn between labels* checkbox in Labeler’s tasks step. Once enabled, you can select the label set for the arrow labeling task. {% hint style="info" %} You can assign both human members and labeling agents. Each model counts toward your assignment limit. {% endhint %} ### Launch the project and trigger labeling When you click **Launch project**, the span labeling agent automatically starts labeling the assigned data. The generated labels can then be reviewed in the project just like other annotations. ### Review labels applied by the labeling agent Once the project has been labeled, reviewers can inspect the annotations to verify: * whether the correct text spans were selected * whether each span was assigned the correct label * whether model output is consistent across similar examples This review step is recommended before submission, export, or downstream use. ### View and compare performance You can track the performance of both human labelers and labeling agents from the [Analytics page](https://docs.datasaur.ai/workspace-management/analytics). From there, you can compare IAA scores and other quality metrics across all labelers, including both humans and models. ### Limitations * Each span can only have one label. * Model performance depends heavily on prompt quality and label definition clarity. # Row Labeling Agent Use a row labeling agent when you want a model to automatically answer [Row Based](https://docs.datasaur.ai/data-studio-projects/nlp-task-types/row-based) questions in a Datasaur project. This is useful for classification, extraction, tagging, scoring, and other structured labeling workflows where the model should generate answers for one or more questions on each row. A row labeling agent works alongside human labelers, so you can compare model output, review quality, and measure performance in the same project. {% hint style="info" %} Models must be deployed in [LLM Labs](https://docs.datasaur.ai/llm-projects/deployment) in the same team workspace as the Data Studio project before they can be assigned as labeling agents. {% endhint %} ### Before you begin Before assigning a row labeling agent, make sure you have: * a deployed model in LLM Labs * a row labeling project in Data Studio * one or more row-labeling questions configured in the project * prompt instructions that clearly define how each question should be answered ### How row labeling agents work For row labeling projects, the labeling agent reads the selected input columns and generates answers for the assigned question scope. A typical setup flow looks like this: {% hint style="info" %} Pre-requisite: [Deployed model in LLM Labs](https://docs.datasaur.ai/agent/labeling-agent) {% endhint %} 1. Assign the deployed model as a labeling agent in Data Studio 2. Launch the project and review the generated answers ### Supported question types The labeling agent supports the following question types in row labeling projects:
Question typeDescription
RadioSingle-select from a predefined list of options
DropdownSingle-select via a dropdown menu
Hierarchical dropdownNested dropdown with parent-child option relationships
TextFree-form text input
DateDate picker input
TimeTime picker input
CheckboxMulti-select from a list of options
SliderNumeric value selection via a slider control
URLText input validated as a URL
### Assign the model as a row labeling agent To assign a deployed model as a labeling agent: 1. Go to **Projects** and click **Create new project**. 2. Upload your files and select **Row Labeling**. 3. Configure Labeler’s tasks 4. In the **Assignment** step, open the **Labeling agents** tab. 5. Select the deployed models you want to assign as labeling agents.

Selecting deployed LLM Labs Sandbox as labeling agent

6. You need to set the agent task by clicking Set a default agent task or Set task scope on individual labeling agent. #### Configure row labeling tasks

Configuring labeling agent tasks - Row Labeling

You can configure: 1. **Target questions**: the one or more questions you want the labeling agent to answer. 2. **Input columns**: the columns the labeling agent should use as context. A single labeling agent can be assigned to **multiple row-labeling questions**. This is useful when: * several related questions should be answered from the same context * one question is a follow-up to another * your workflow includes conditional question logic * you want one agent to handle a complete question group instead of splitting it across multiple agents {% hint style="info" %} You can assign both human members and labeling agents. Each model counts toward your assignment limit. {% endhint %} ### Multiple-question assignment In row labeling projects, one labeling agent can answer more than one question. This is helpful when the questions are related and should be answered from the same input context. For example, one agent can answer: * the primary classification question * a follow-up detail question * a supporting explanation or extraction field Using one agent for related questions can improve consistency because the model sees the same context and can reason across the connected questions in one workflow. #### Conditional question workflows Multiple-question assignment is especially useful for **conditional question workflows**. For example: * **Question 1:** Is this feedback about billing? * **Question 2:** If yes, what is the billing issue type? If both questions are assigned to the same labeling agent, the agent can use the answer to the first question as pre-requisite condition for the second. This is useful for: * decision-tree style annotation * follow-up classification * structured extraction with branching logic * survey or form-style review workflows {% hint style="info" %} For best results, explicitly describe the conditional logic in your model instructions, including when the agent should answer a follow-up question and when it should skip it. {% endhint %} ### Launch the project and trigger labeling When you click **Launch project**, the row labeling agent automatically starts labeling the assigned data. If a labeling agent has been assigned multiple row-labeling questions, it will generate answers for all questions within its configured task scope. ### Review answers applied by the labeling agent Once all questions are fully answered, either through external model assistance or manual input, the project can undergo a final review. This stage typically involves a reviewer ensuring the consistency and accuracy of all annotations before submission or export through reviewer mode. ### View and compare performance You can track the performance of both human labelers and labeling agents from the [Analytics page](https://docs.datasaur.ai/workspace-management/analytics). From there, you can compare IAA scores and other quality metrics across all labelers, including both humans and models. ### Limitations * Limited supported question type for optimized performance. * Model performance depends heavily on prompt quality and label definition clarity. # LLM Labs Introduction LLM Labs is where teams build, compare and evaluate private and open-weight LLMs before deploying them inside their own infrastructure.
Welcome to Datasaur’s LLM Labs! This guide will introduce you to the key features and tools available in our LLM Labs, designed to enhance your projects. Whether you're a new user or looking to explore more advanced functionalities, this documentation will help you get started. Datasaur offers a **free tier** for new users in a trial period. When you sign up, you will receive a credit to try out our powerful LLM Labs. This allows you to explore various features and assess how they can benefit your projects without any initial cost. In LLM Labs, all activities are organized within a workspace. The workspace is where you perform tasks related to model evaluation, dataset management, knowledge base creation, and model catalog exploration. It supports a collaborative environment, allowing you to invite team members to work together seamlessly. This collaborative setup ensures that all project-related activities and resources are easily accessible and manageable in one place. ### Key features Datasaur LLM Labs empowers you with: * [**Sandbox**](https://docs.datasaur.ai/llm-projects/sandbox)**:** Experiment and interact with various LLMs directly within Datasaur. * [**Knowledge base**](https://docs.datasaur.ai/llm-projects/vector-store)**:** Store and query text embeddings for tasks like semantic search and retrieval augmented generation (RAG). * [**Evaluation (Rating and Ranking)**](https://docs.datasaur.ai/llm-projects/evaluation)**:** Evaluate LLM outputs through human feedback by rating and ranking responses based on quality, accuracy, etc. * [**Automated Evaluation**](https://docs.datasaur.ai/llm-projects/evaluation/automated-evaluation)**:** Set up automatic scoring of LLM outputs against predefined metrics, saving you time and effort in evaluation. ### **Still Have Questions?** If you're still unsure about how to use LLM Labs or have any questions, please don't hesitate to reach out to our support team at . We're here to help you get the most out of our platform and achieve your LLM development goals. # Sandbox Sandbox is a platform designed for your LLM experimentation. It allows you to integrate your knowledge base and deploy models easily. ## Overview Sandbox is a key feature within LLM Labs, that provides a user-friendly environment specifically designed for LLM experimentation. It allows you to: * **Connect your preferred base models:** Integrate your choice of base models to explore their functionalities. * **Create a dedicated sandbox:** Set up a personalized workspace for your LLM experimentation. * **Configure your model:** This configuration process defines your models. It involves defining elements like: * **Instruction:** Craft an instruction that specifies the format for user prompts sent to your model. This ensures consistency and clarity in user interactions. * **Context (knowledge base):** Optionally, integrate a context knowledge base to provide additional background information to the model, potentially improving its understanding and response accuracy. * **Configuration:** Fine-tune various parameters for the connected models, such as temperature or token settings, to optimize its performance for your specific use case. * **Run prompts:** Test your models with various prompts to evaluate their responses and refine your approach. Sandbox offers several advantages for LLM enthusiasts and developers: * **Reduced risk:** Experiment with different models without committing to deployment, minimizing potential risks associated with real-world use cases. * **Enhanced understanding:** Gain deeper insights into individual model and their capabilities through hands-on experimentation. * **Optimized configuration:** Fine-tune model parameters within the sandbox to achieve the best possible results for your specific needs. * **Streamlined development:** Test and refine your models in a controlled environment before deployment, ensuring optimal performance. ## Quick guide Sandbox is designed for ease of use. Here's a quick guide to get you started: ### Step 1: Create your Sandbox
* Click **Create new sandbox**. * You can rename your sandbox to make it easier to identify. ### Step 2: Configure the model
* In the model you want to configure, click **Change model settings** from the three-dot menu. * You can adjust various parameters for your models, such as temperature or token settings. * Experiment with different configurations to observe their impact on the base model's responses. Learn more about [base models](/llm-projects/models). ### Step 3: Run prompts
* Enter your desired prompts within the designated area. * This prompt can be a question, a task instruction, or any text input you want the model to process. * Click **Run selected** to to trigger the models to generate responses to your prompts. ## Deploy models {% hint style="info" %} Learn more about deploying your models on this [page](/llm-projects/deployment). {% endhint %} ## Save to library The **Save to library** feature allows you to save models from a sandbox. This enables you to reuse these models across various features, such as evaluation, without needing to rebuild them from scratch. {% hint style="info" %} **Save to library** feature can be used by users within the same workspace, meaning you can share your models with other team members in that workspace. {% endhint %} ### Save a model 1. To save your model, click the **Save to library** option from the model's three-dot menu.
2. After clicking **Save to library**, give your model a descriptive name, and click **Save model** to save it to the library.
### Access the saved models in Sandbox 1. In the **Models** section, click **Select from library** from the three-dot menu.
2. Choose the saved model that you want to use, and click **Select model**.
### Access the saved models in Evaluation 1. Navigate to the **Evaluation** page under LLM Labs menu. 2. Click **Create evaluation project** and choose **Automated evaluation** project type.
3. In the **Model** dropdown, select the saved models you want to evaluate.
4. Continue the creation process for automated evaluation. Learn more on how to create an [automated evaluation project](/llm-projects/evaluation/automated-evaluation#getting-started). ## Add prompts from dataset In Sandbox, you can efficiently test your models against multiple inputs by importing prompts directly from an existing dataset. {% hint style="info" %} Before you can use this feature, ensure that you have at least one dataset available in your LLM Labs project. Learn more on how to [create a dataset](/llm-projects/dataset). {% endhint %} To add prompts from dataset: 1. In the **Prompts** section, click **Add prompts from dataset** from the three-dot menu.
2. After selecting this option, you will be able to choose a dataset and map its columns to be used as prompts. ## Model configuration
### Basic hyperparameters You can configure your models with your desired settings. Several parameters that you can adjust based on your needs are: 1. **Temperature**: This parameter controls the randomness of the generated text. A higher temperature will result in more diverse and creative responses, while a lower temperature will produce more focused and predictable responses. 2. **Top P**: This parameter, also known as nucleus sampling, controls the cumulative probability threshold for token selection. A higher Top P value will result in more focused and relevant responses, while a lower Top P value will allow for more diverse and unexpected responses. 3. **Maximum output tokens**: This parameter sets the maximum number of tokens that will be generated in the response. A higher maximum length will allow for longer and more detailed responses, while a lower maximum length will result in shorter and more concise responses. 4. **Maximum knowledge base tokens**: This parameter, defines the upper limit on the number of tokens that can be stored in the knowledge base. This is crucial for maintaining efficient storage and retrieval times. It ensures that the knowledge base doesn't become overloaded with data, which can slow down queries and decrease performance. 5. **Similarity score**: This parameter controls how closely the generated text matches the original prompt in terms of content and style. A higher similarity score will result in responses that more closely align with the prompt, while a lower similarity score will allow for more divergent responses. ### Advanced hyperparameters Advanced hyperparameters in Sandbox lets you customize your model behavior further by adding additional parameters in JSON format. This gives you greater control over the underlying model's response generation and can optimize performance for specific use cases. To configure advanced hyperparameters: 1. In the model you want to configure, click **Change model settings** from the thrree-dot menu.
2. Scroll down to the **Advanced hyperparameters** accordion.
3. Enter your desired hyperparameters in valid JSON format. For example: ``` { "stream": true } ``` ### Anonymize PII The **Anonymize PII** feature enhances the privacy and security of sensitive data during inferencing in the Sandbox. This feature is designed to automatically detect and mask Personally Identifiable Information (PII), ensuring that sensitive details remain confidential while retaining readability and context in the output. #### How it works When users submit prompts containing PII in the Sandbox, the **Anonymize PII** feature securely masks the sensitive information in a structured format. For instance, names, organizations, and locations are replaced with easily identifiable placeholders. These placeholders make it clear that data has been redacted without losing the context required for meaningful results. Example: * Original input: ``` John Doe works at Datasaur Software and attends meetings with Innova Aspire Inc. ``` * Masked output: ``` [PERSON#0] works at [ORGANIZATION#0] and attends meetings with [ORGANIZATION#1]. ``` This ensures consistency, especially when the same PII subject appears multiple times in the same prompt. The system assigns unique identifiers for different entities, avoiding confusion caused by generic replacements. #### How to use 1. In the model you want to configure, click **Change model settings** from the three-dot menu. 2. Enable **Anonymize PII (English only)**.
3. Run your prompt as usual, and all PII in the output will be automatically masked.
## Auto-generate instruction (BETA) **Auto-generate instruction** feature in LLM Labs is designed to help you in generating better-structured system instructions in the Sandbox. This tool enables you to generate customized instructions for your models, making it easier to define the assistant's behavior and optimize responses. {% hint style="info" %} * This feature is currently in **BETA** and uses **OpenAI GPT-4o** for instruction generation, providing robust assistance tailored to various models. * Users are encouraged to refine and double check generated instructions to meet specific needs, as auto-generated content may sometimes require minor adjustments for optimal results. {% endhint %} To auto-generate instructions: 1. Click the **Help me write** button in a model.
2. In the **Auto-generate instruction** dialog, describe what you’d like the model to achieve.
{% hint style="info" %} You may also select from example instructions such as `Write, debug, and optimize my code` to guide the auto-generation process. {% endhint %} 4. Click **Generate instruction** button to receive a structured system instruction based on your input. Review the generated instruction to ensure it matches your intended use case.
5. Click the **Use this instruction** button to add the system instruction into your model. ## Cost prediction calculation Cost prediction provides transparency and predictability in costs. You can see how much your inference activities will cost in LLM Labs. To see cost prediction: 1. Open your sandbox. 2. Write your prompts, and the predicted cost for each prompt will appear.
{% hint style="info" %} The cost prediction will **calculate the cost based on your available models.** The more models you have, the higher the cost will be. {% endhint %} To see the prediction details, click the **View predicted cost** option from the three-dot menu of each prompt.
It will show you a dialog of the detailed cost. You can also break down and compare the cost prediction based on your available models.
# Direct Access LLMs ## Overview Datasaur offers Direct Access LLMs, which let you instantly use the most popular Large Language Models (LLMs) within the platform. This feature eliminates the need for complex API key setup and multi-cloud configurations. Additionally, you can cut wait lines and immediately access the latest state of the art models. ## Supported providers Datasaur's Direct Access LLM includes models from Microsoft, OpenAI, Amazon, and Google. The sections below describe each provider and the models they offer. ### Azure OpenAI

Azure OpenAI

With Azure OpenAI, you can utilize the following models: * **gpt-4o**: A highly advanced model boasting an expansive knowledge base for richer and more comprehensive responses. * **gpt-4-32k**: A variant of the gpt-4 model, with greater capacity to handle longer inputs. * **gpt-4 turbo**: A high-performance model optimized for speed and efficiency. * **gpt-4**: A powerful model offering advanced language understanding and generation capabilities. * **gpt-35-turbo-16k**: A variant of the gpt-35-turbo model, with greater capacity to handle longer inputs. * **gpt-35-turbo**: A fast and efficient model ideal for applications requiring rapid response times. ### OpenAI

OpenAI

With OpenAI, you can utilize the following model: * **o1-mini-2024-09-12:** A compact and efficient model well-suited for tasks that require fast inference and lower computational resources. It excels in short-form text generation, question answering, and text summarization. * **o1-preview-2024-09-12:** A preview model offering advanced capabilities and access to the latest developments in OpenAI's LLM technology. This model is ideal for exploring cutting-edge language processing tasks and experimenting with potential future functionalities. * **gpt-4o-mini**: A streamlined and efficient version of the advanced gpt-4o model, designed to deliver rich responses while requiring less computational power. * **gpt-4o**: A highly advanced model boasting an expansive knowledge base for richer and more comprehensive responses. * **gpt-4 turbo**: A high-performance model optimized for speed and efficiency. * **gpt-4**: A powerful model offering advanced language understanding and generation capabilities. * **gpt-35-turbo-16k**: A variant of the gpt-35-turbo model, with greater capacity to handle longer inputs. * **gpt-35-turbo**: A fast and efficient model ideal for applications requiring rapid response times. ### Amazon Bedrock

Amazon Bedrock

With Amazon Bedrock, Datasaur provides several open source models, such as: * **Claude 3.5 Sonnet**: An enhanced version of Claude 3 Sonnet, with updated knowledge and improved reasoning capabilities. * **Claude 3 Sonnet**: A more verbose Claude model, offering deeper analysis and extended conversations. * **Claude 3 Opus**: The most comprehensive Claude model, providing in-depth expertise across a wide range of subjects. * **Claude 3 Haiku**: A concise and efficient AI assistant, perfect for brief, focused interactions. * **Claude 2.1**: An updated version of Claude 2.0, featuring refinements in language understanding and generation. * **Claude 2.0**: An upgraded Claude model with expanded knowledge and improved conversational abilities. * **Claude Instant**: A rapid-response AI assistant for quick, concise interactions. * **Meta Llama 3 70b Instruct**: A variant of the Meta Llama 3 8b Instruct model, with increased capacity and performance. * **Meta Llama 3 8b Instruct**: A newer model optimized for instruction-following tasks, offering high accuracy and reliability. * **Meta Llama 2 Chat 70B**: A variant of the Meta Llama 2 Chat 13B model, with increased capacity and performance. * **Meta Llama 2 Chat 13B**: A highly advanced model designed for conversational AI applications. * **Mistral Large**: A more expansive version of Mistral, offering deeper knowledge and more nuanced interactions. * **Mixtral 8x7B Instruct**: An advanced instruction-following model combining multiple expert systems for enhanced performance. * **Mistral 7B Instruct**: A compact yet powerful model designed for following instructions with precision. * **Mistral Small**: A nimble AI assistant optimized for quick responses and everyday tasks. * **Command R+**: The most advanced Command model, featuring superior problem-solving and creative abilities. * **Command R**: An enhanced version of Command, with improved reasoning and analytical skills. * **Command**: A versatile AI assistant balancing speed and capability for various applications. * **Command Light**: A streamlined AI model for efficient, straightforward task completion. * **Amazon Titan Text Premier**: Amazon's most advanced text AI, offering sophisticated language understanding and generation. * **Amazon Titan Text Express**: A mid-range AI assistant balancing efficiency and capability for various text-based applications. * **Amazon Titan Text Lite**: A lightweight AI model for basic text processing and generation tasks. ### Vertex AI

Vertex AI

With Vertex AI, you can utilize the following models: * **Gemini 1.5 Pro**: A high-performance model offering advanced language understanding and generation capabilities. * **Gemini 1.5 Flash**: A variant of the Gemini 1.0 Pro model, optimized for speed and efficiency. * **Gemini 1.0 Pro**: A highly advanced model offering exceptional language understanding and generation capabilities. ### Azure AI

Azure AI

With Azure AI, you can utilize the following models: * **Meta-Llama-3-1-405B-Instruct**: A massive 405 billion parameter version of Meta-Llama hosted on Azure AI, specifically optimized for following instructions and demonstrating exceptional proficiency in complex language understanding and generation. * **Meta-Llama-3-1-70B-Instruct:** A powerful 70 billion parameter model hosted on Azure AI, fine-tuned for instruction following. This model offers a balance between scale and efficiency, making it well-suited for a wide range of LLM tasks. * **Meta-Llama-3-1-8B-Instruct:** An efficient 8 billion parameter version of Meta-Llama on Azure AI, suitable for tasks where resource efficiency is crucial while maintaining respectable language processing capabilities. ### Hugging Face

Hugging Face

With Hugging Face, you can utilize the following models: * **Meta-Llama-3.1-70B-Instruct**: A powerful 70 billion parameter model from Meta fine-tuned for following instructions. This model excels at complex language tasks, text generation, question answering, and code generation. * **Meta-Llama-3.1-8B-Instruct:** A smaller 8 billion parameter version of Meta-Llama, offering a good balance between performance and efficiency. It is suitable for tasks where resource constraints are a factor while still maintaining good language understanding and generation capabilities. * **Mistral-7B-Instruct-v0.1:** The first iteration of the Mistral-7B model fine-tuned for instruction following. This model excels in code generation, reasoning tasks, and understanding complex instructions. * **Mistral-7B-Instruct-v0.2:** An improved version of the Mistral-7B-Instruct model with enhanced instruction-following capabilities and performance. * **Mistral-7B-Instruct-v0.3:** The latest iteration of the Mistral-7B-Instruct model, further refined for better accuracy, coherence, and instruction adherence in text generation tasks. * **Mistral-Nemo-Instruct-2407:** A specialized version of the Mistral model trained on a massive dataset for improved code generation and technical language understanding. This model is particularly useful for tasks involving code-related data or highly technical language. * **Mixtral-8x7B-Instruct-v0.1:** A powerful model combining eight smaller 7 billion parameter models for enhanced performance and capabilities. This model showcases advancements in mixture-of-experts architecture, demonstrating strong performance in various natural language processing tasks. ## Use in Sandbox To get started with Direct Access LLMs, navigate to the [Sandbox](/llm-projects/sandbox) page and create a new Sandbox. Once the Sandbox is created, you can change the base model to any model from Direct Access LLMs.
# File attachment ## Overview File attachments lets you upload files or URLs as part of your prompt inputs when interacting with the model. This feature enhances the prompting experience by providing additional context, data, or reference materials that the model can use to generate more accurate and relevant responses. {% hint style="info" %} File attachments provide additional context or reference materials. Unlike the knowledge base or Retrieval-Augmented Generation (RAG), which send only relevant portions, all attached files are sent to the model for a more comprehensive input. {% endhint %} ## Get started {% hint style="info" %} Our file attachment feature supports `.png`, `.jpg`, `.jpeg`, `.webp`, `.gif`, `.pdf`, `.csv`, and `.txt` formats. You can upload up to **5 files per prompt**, with a maximum size of **50 MB** each. {% endhint %} 1. Open your Sandbox and click the attach button on a prompt. You can attach files or URLs.
2. After attaching a file, enter an additional prompt or leave it blank. Then, click **Run selected**.
{% hint style="info" %} Please note that the attached files and URLs will be visible to all workspace members and will remain accessible until deleted. {% endhint %} ## View or delete attachments 1. To delete an attachment, hover over the item and click the `X` button.
2. To view an attachment, click the item.
# Conversational prompt ## Overview Conversational prompting supports multi-turn conversation testing in LLM Labs. It allows prompts and responses to be evaluated across multiple interactions, helping assess how well a language model maintains context and coherence throughout a conversation. This feature can be used to test and analyze model behavior in conversational workflows. ## Get started 1. Navigate to your [Sandbox](/llm-projects/sandbox). 2. Click the three-dot menu in a prompt.
3. Choose your method: * **Add conversation**: Manually input a new conversation directly into the prompt. Learn more on how to [write the conversation](#write-conversation-manually).
* **Upload conversation file**: Upload a `.json` conversation file for testing. Learn more about the [JSON file format](#create-the-json-file-format).
4. Execute the conversation and observe how the language model performs, maintaining context and coherence across turns.
### Understanding roles Roles are used to represent the conversational input from the user and the assistant. * **User:** Represents input from the user, such as a request or question sent to the model. * **Assistant:** Represents the model's response, either as a reply or part of the conversation history in a prompt. You can manually change the roles via the dropdown when writing the conversational prompt to define the prompt input. This flexibility allows you to simulate various conversational dynamics and test how the model responds to different types of interactions.
### Write conversation manually 1. In the **Add conversation** dialog:
1. Write the prompt to represent the input based on your needs, whether it is for the assistant or user roles. 2. Use the role dropdown to specify the role for each message (user or assistant). 3. Enter the content of the message in the provided text area. 2. Add more messages to the conversation: You can also add more messages in this dialog. To add a message, hover to the text input, and there will be 3 buttons: 1. **Add above:** When clicked, a new text input will appear above the current text input that you are hovering. 2. **Add below:** When clicked, a new text input will appear under the current text input that you are hovering. 3. **Delete:** When clicked, it will delete the message. 3. **Reorder the conversation**: You can drag and drop each message to reorder the conversation.
4. **Attach files and URLs**: You can attach files or URLs to messages with the **User** role. This capability is useful for simulating real-world scenarios where users might share documents or links. Learn more about [file attachment](/llm-projects/sandbox/file-attachment).
{% hint style="info" %} Each message in a conversation can have up to 5 attachments. For example, with 3 messages in a prompt, you can have up to 15 attachments. {% endhint %} ### Create the JSON file format **Structure:** The core structure is an array of objects: ```jsx [ { "content": "...", "role": "..." }, { "content": "...", "role": "..." }, // ... more turns ] ``` **Fields:** * **`content`:** * This field holds the actual message or interaction data. * It can be a simple string for plain text messages. * Alternatively, it can be an array of objects to include attachments like URLs or images. * This field **cannot be empty**. Every conversation turn must contain content. * **`role`:** * This field indicates the speaker's role in the conversation. * It can be either `"user"` or `"assistant"`. **Attachments (URLs and files):** To include attachments, the `content` field should be an array of objects. Each object within this array can represent text, a file URL, or a website URL. * **Text:** * Use `{"type": "text", "text": "Your text here"}`. * **File URL:** * Use `{"type": "image_url", "image_url": {"url": ""}}`. * **URL (Website link):** * Use `{"type": "image_url", "url": ""}`. **Sample JSON file**: {% file src="/files/WIRTVv1SALXms5N2F6KW" %} ## Conversational vs basic prompt
#### Conversational prompt * Maintains and understands the full history of the conversation. * Handles back-and-forth dialogue naturally. * Generates replies based on accumulated context. #### Basic prompt * Treats each input as an independent request. * Has no memory of previous interactions. * Generates replies based solely on the immediate input. # Model Context Protocol (MCP) ### Overview [Model Context Protocol](https://modelcontextprotocol.io/introduction) (MCP) is an open protocol that standardizes how applications provide tools and context to LLMs. This enables LLMs to go beyond answering questions by allowing them to interact with external tools, retrieve relevant context on demand, and perform complex, multi-step tasks. MCP transforms LLMs from passive responders into active agents capable of tool use, in-application reasoning, and dynamic task execution. This enhances their ability to assist with workflows, automate decisions, and collaborate more effectively within real-world systems. This document assumes you are already familiar with the concept of Tool Calling and Model Context Protocol. If you need more information about these topics, you can read the links below: * [OpenAI Tool Calling](https://platform.openai.com/docs/guides/function-calling?api-mode=chat) * [Model Context Protocol: Concept](https://modelcontextprotocol.io/docs/concepts/architecture) * [Model Context Protocol: Transports](https://modelcontextprotocol.io/specification/2025-06-18/basic/transports) This document outlines LLM Labs' support for MCP servers and how to connect them. You'll learn how to configure the server, set up secure communication, and enable the integration to begin managing tools, context, and workflows through a standardized interface. {% hint style="info" %} Please note that we are not affiliated with any of the tools or MCP servers mentioned in the document. They are included for demonstration purpose only, and their use falls outside our terms of service. They should be used at your own discretion. {% endhint %} ### LLM Labs' support for MCP servers #### **Supported Transports** LLM Labs can connect to **remote MCP servers**: MCP servers that implement the following transports: * [**Streamable HTTP**](https://modelcontextprotocol.io/specification/2025-06-18/basic/transports#streamable-http) * [**HTTP with Server-Sent Events (SSE)**](https://modelcontextprotocol.io/specification/2024-11-05/basic/transports#http-with-sse) *(deprecated)* Support for MCP servers with [stdio](https://modelcontextprotocol.io/specification/2025-06-18/basic/transports#stdio) transport is achievable with the help of additional tool to adapt stdio MCP servers as a remote MCP server. There are several open source tools available that helps converting MCP servers, such as [Supergateway](https://github.com/supercorp-ai/supergateway). #### Supported authentication methods Remote MCP servers implement different levels of authentication method, such as: open access (no authentication), OAuth, and API key-based authentication through `Authorization` header. LLM Labs supports open access and API key-based authentication, with OAuth support planned for a future release. #### Supported MCP Server Features As written in the [specification](https://modelcontextprotocol.io/specification/2025-06-18/server/index), there are 3 main features: **Prompts**, **Resources**, and **Tools**. LLM Labs only supports the **Tools** feature at this moment, as we believe that this feature is the most broadly used. ### Connecting your LLM Labs application to remote MCP servers #### Configuring LLM Labs application We can configure the LLM Labs application as an MCP client that can connect remote MCP servers. Follow the steps below: 1. Go to [Sandbox](https://docs.datasaur.ai/llm-projects/sandbox) and open an existing one or create a new one. 2. Open the desired application to connect to a remote MCP server, or create a new application in a sandbox. 3. Ensure that the selected LLM model supports tool calling feature. 4. Register the desired MCP servers via [Advanced hyperparameters](https://docs.datasaur.ai/llm-projects/sandbox#advanced-hyperparameters) (The gear icon in an application) You can configure the server under `datasaur.mcpServer.[mcp_server_name]` with the following fields: * `url` (required): The URL of the MCP server. * `headers` (optional): Headers for the connection. **Example 1: No authentication** ```jsx { /* ... other configuration ... */ "datasaur": { "mcpServers": { "fetch": { "url": "" } } } } ``` **Example 2: With API key via authorization header** ```jsx { /* ... other configuration ... */ "datasaur": { "mcpServers": { "stripe": { "url": "", "headers": { "Authorization": "Bearer TOKEN" } } } } } ``` 5. After configuring the application, you can run it against some prompts. It will automatically try to access the tools provided when running the prompt.
#### Consuming the MCP call outputs in your application Our deployed application conform with OpenAI Chat Completion API specification that doesn’t natively support MCP call outputs. OpenAI supports the MCP call outputs via their new Response API. We are working to support the Response API specification for deployed applications, and will be available soon. While we are working on the Response API support, we also extend the OpenAI Chat Completion API’s Message Object with an additional `events` field, with the following components: * **type**: The type of event. There are three types of events for Tool Runner Calls: * `tool_runner_call`: Triggered when the model calls a tool. It includes the arguments passed to the tool. * `tool_runner_call_result`: Triggered when a tool runs successfully. It contains the tool's execution result. * `tool_runner_call_error`: Triggered when a tool fails during execution. It includes error information. * **data**: The event's payload. * **content\_at**: The character index in the content where the event occurred. ### Alternative approach: connecting to MCP servers during runtime While configuring it directly from the application is the most common approach for MCP server integration, there are some cases where you may need to connect MCP servers dynamically from your system when calling the deployed application endpoint. In this case, you can send the MCP Servers configuration directly by specifying it as a tool similar to the [OpenAI's Remote MCP Interface](https://platform.openai.com/docs/guides/tools-remote-mcp#page-top), for example: ```jsx { "messages": [{ "role": "user", "content": "Create a payment link for me." }], "tools": [ { "type": "mcp", "server_label": "stripe", "server_url": "", "headers": { "Authorization": "Bearer {STRIPE_API_KEY}" } } ] } ``` * **Parameters**: * `server_label` (required): The name of the MCP Server. * `server_url` (required): The URL of the MCP Server. * `headers` (optional): Headers for the connection. We hope this guide helps you get started with connecting MCP servers to LLM Labs. As we continue to expand support and improve integration capabilities, your feedback is always welcome. If you have any questions or run into issues, please don't hesitate to reach out to our team. # Deployment ### Overview The deployed models page in LLM Labs provides a centralized place to manage all deployed model APIs efficiently. It serves as a hub where users can oversee their deployed models, track their status, and perform necessary maintenance operations. By offering a user-friendly interface, this feature streamlines the deployment workflow, making it simple to monitor and manage models at scale. ### Get started {% hint style="info" %} To deploy a model, you will need to create a Sandbox first. [Learn more about Sandbox](https://docs.datasaur.ai/llm-projects/sandbox). {% endhint %} 1. After creating and configuring your model on the Sandbox page, click the **Deploy** button.
2. A dialog box will appear, allowing you to review and name your model before deployment. Once you have reviewed the details and are ready to proceed, click the **Deploy model** button.
3. Once you’ve deployed the model, you will be redirected to the deployment details page. Here you can: 1. **Create API Key**: Generate an API key for use when calling the API endpoint. 2. **Suspend**: Temporarily pause the API endpoint. Systems using these suspended endpoints may not function until it is resumed. 3. **Delete**: Delete the deployment. Deleting a deployment permanently removes its configurations and active endpoints. Any systems using these endpoints will stop functioning properly. 4. **Use in Sandbox**: Choose the sandbox environment where you want to create a copy of the deployed model for iteration. After making changes, you can redeploy it to the same deployment. 5. Review the statistics of the deployment.
4. You can also manage your list of previously created deployments on the **Deployed models** page.
{% hint style="info" %} To learn more about how to use our Deployment API, please refer to this [Deployment API](https://docs.datasaur.ai/llm-projects/deployment/deployment-api) page. {% endhint %} # Deployment API ## Overview This page explains how to use our Deployment API with several use cases, providing detailed instructions and examples for different scenarios. ## Run Deployed LLM Application with Chat Completion Executes a RAG-enhanced chat completion request against a deployed LLM application. ### Endpoint ``` POST https://deployment.datasaur.ai/api/deployment/:teamId/:deploymentId/chat/completions ``` **Path Parameters** | Parameter | Type | Description | | ------------ | ------ | ----------------------------------------- | | teamId | string | Your team identifier | | deploymentId | string | The ID of your LLM application deployment | **Query Parameters** | Parameter | Type | Description | | --------- | ------ | ----------------------------- | | source | string | (Optional) Source identifier | | sourceId | string | (Optional) Specific source ID | ### Request Body The request body follows the OpenAI-compatible chat completion format, with additional RAG-specific enhancements. **Key Properties** | Property | Type | Required | Description | | ---------------- | ---------------- | -------- | ---------------------------------------------------------------------------------- | | messages | array | Yes | Array of message objects representing the conversation | | stream | boolean | No | Enable streaming responses | | include\_usage | boolean | No | | | rewrite\_query | string | No | Override the behavior of summarizing the messages as query with an arbitrary text. | | filter\_metadata | object \| string | No | Filter data based on the File Properties attached to each file. | **Message Types** Messages can be of the following roles: * `user`: User inputs * `assistant`: Assistant responses * `tool`: Tool/function responses {% hint style="info" %} This API doesn’t support the `system` role and will inherit the System Instruction from the sandbox. See: {% endhint %} ## Advanced Features ### Multimedia Content Types The `content` field in messages supports various types: ```typescript type ContentPart = | TextContent | ImageContent | URLContent; ``` #### **Text Content** ```json { "type": "text", "text": "Your text here" } ``` #### **Image Content** ```json { "type": "image", "image_url": { "url": "https://example.com/image.jpg", "detail": "high" } } ``` #### **URL Content** ```json { "type": "url", "url": "https://example.com", "name": "Optional name", "options": { "select_pages": "1-3", "include_page_screenshot_as_image": false } } ``` The URL content type supports both standard web URLs and base64-encoded data URLs, allowing you to: 1. **Reference external web content**: Use standard URLs () 2. **Embed file content directly**: Use base64 data URLs for PDF, HTML, or other content types ``` "url": "data:application/pdf;base64,JVBERi0xLjMKJcTl8uXrp..." ``` **Options:** * **select\_pages**: Specifies which pages to process from multi-page documents (PDFs) * Format: "1-5" (range), "1,3,5" (specific pages), or "1-3,7,9-11" (combination) * Example: "select\_pages": "1-3,5,8-10" * Default: All pages if not specified * **include\_page\_screenshot\_as\_image**: When set to true, includes visual representations of pages. The visual representations are also sent to the model when the model selected in the sandbox application supports visual capability. * For PDFs: Renders page as image for visual analysis * For websites: Captures screenshot of the rendered page * Enables the model to analyze visual layouts, charts, and non-text elements * Default: false Base64 encoding is particularly useful for: * Embedding content directly without requiring separate file uploads * Processing temporary or dynamically generated content * Working with content that doesn't have a public URL ### Advanced Retrieval: Query Rewriting Query rewriting is a technique used in information retrieval and search systems to modify or enhance the original search query to improve search results. There might be cases where you prefer a customized way to summarize the message. You can rewrite the query with your version by specifying the `rewrite_query`. For example: ```json { "messages": [ { "role": "user", "content": "What's the weather in Bali?" }, { "role": "assistant", "content": "It's hot and humid." }, { "role": "user", "content": "How about in Jakarta?" } ], "rewrite_query": "What's the weather in Jakarta?" } ``` ### Advanced Retrieval: Metadata Filtering Another way to improve the accuracy of the retrieval is through additional filtering, to ensure only relevant information is retrieved. Below is an example of filtering search result based on the `jurisdiction` and the `date`. ```json { "filter_metadata": { "bool" : { "must" : [ { "bool" : { "should" : [ { "term" : { "jurisdiction" : "alabama" } }, { "term" : { "jurisdiction" : "florida" } }, { "term" : { "jurisdiction" : "nevada" } } ] } }, { "range" : { "date" : {"gt": "2024-10-29" } } } ] } } } ``` ## Example Usage ### Simple Text Query ```json { "messages": [ { "role": "user", "content": "What is the capital of France?" } ] } ``` ### Multi-turn Conversation ```json { "messages": [ { "role": "system", "content": "You are a helpful assistant who specializes in geography." }, { "role": "user", "content": "What is the capital of France?" }, { "role": "assistant", "content": "The capital of France is Paris. It's often called the 'City of Light' (Ville Lumière)." }, { "role": "user", "content": "Tell me more about its population." } ] } ``` ### Text with Images ```json { "messages": [ { "role": "user", "content": [ { "type": "text", "text": "What's in this image?" }, { "type": "image_url", "image_url": { "url": "https://example.com/image.jpg", "detail": "high" } } ] } ] } ``` ### Text with PDF Analysis ```json { "messages": [ { "role": "user", "content": [ { "type": "text", "text": "Please summarize this research paper" }, { "type": "url", "url": "https://doompdf.pages.dev/doom.pdf", "options": { "select_pages": "1-5", "include_page_screenshot_as_image": true } } ] } ] } ``` ### URL with Base64 Screenshot ```json { "messages": [ { "role": "user", "content": [ { "type": "text", "text": "What does this webpage contain?" }, { "type": "url", "url": "data:text/html;base64,SGVsbG8gV29ybGQ=", "name": "Example Page", "options": { "include_page_screenshot_as_image": true } } ] } ] } ``` ### Metadata Filtering ```json { "messages": [ { "role": "user", "content": [ { "type": "text", "text": "What's in pages 2-5 of the technical documentation?" } ] } ], "filter_metadata": { "bool": { "must": [ { "bool": { "should": [ { "term": { "jurisdiction": "alabama" } }, { "term": { "jurisdiction": "florida" } }, { "term": { "jurisdiction": "nevada" } } ] } }, { "range": { "date": { "gt": "2024-10-29" } } } ] } } } ``` {% hint style="info" %} Each example can be enhanced with additional parameters such as: * `stream: true` for streaming responses * `tools` for function calling capabilities * `rewrite_query` for optimized RAG queries {% endhint %} ## Response ### Non-streaming Response ```typescript { id: string; choices: Array<{ message: { role: 'assistant'; content: string; tool_calls?: Array; }; finish_reason: 'stop' | 'length' | 'tool_calls' | 'content_filter'; index: number; }>; usage: { prompt_tokens: number; completion_tokens: number; total_tokens: number; prompt_embedding_tokens?: number; }; contexts?: Array; } ``` ### Streaming Response Sends chunks of the response as Server-Sent Events (SSE) with the following format: ```json { id: string; choices: Array<{ delta: { content?: string; role?: 'assistant'; tool_calls?: Array; }; finish_reason: string | null; index: number; }>; usage: null | UsageInfo; contexts?: Array | null; } ``` ## Error Handling The API uses standard HTTP status codes with specific actions for resolution: ### 400 Bad Request **Cause**: Invalid parameters or malformed request **Recommended resolution:** * Check request body format and required fields * Validate parameter types and values * Ensure message array is not empty * Check if URLs are properly formatted and accessible ### 401 Unauthorized **Cause**: Missing or invalid authentication **Recommended resolution:** * Check if API key is included in the request header * Verify API key is valid and not expired * Ensure API key has correct format * Generate a new API key if necessary * Contact if API key should be valid ### 403 Forbidden **Cause**: Insufficient permissions for the requested operation **Recommended resolution:** * Verify team membership and permissions * Check if you have access to the specified LLM application * Ensure your subscription covers the requested features * Request necessary permissions from team admin * Upgrade subscription tier if needed ### 429 Too Many Requests **Cause**: Rate limit exceeded, see [Rate Limiting](https://docs.datasaur.ai/llm-projects/deployment/deployment-api#rate-limiting) section below for more details **Recommended resolution:** * Implement exponential backoff retry logic * Check rate limits in response headers * Reduce request frequency * Consider upgrading your plan for higher limits * Optimize batch operations to reduce API calls ### 500 Internal Server Error **Cause**: Server-side error **Recommended resolution:** * Retry request after a brief delay * Check [system status page](https://status.datasaur.ai) for outages * Verify request payload size is within limits * Contact if error persists * Save error response for troubleshooting For all errors, the response will include a detailed error message to help diagnose the issue. If problems persist after taking the recommended actions, please contact with the error details. ## Rate Limiting The API enforces the following rate limits. If any of these limits is reached, subsequent requests will be rejected with a 429 (Too Many Requests) status code until the limit resets: * Origin IP address-based: 1500 requests per 60 seconds * Deployment ID-based: 300 requests per 60 seconds * Team-based: Daily limits apply for free trial accounts Note: These limits are evaluated independently - hitting any single limit will result in request rejection, regardless of the status of other limits. ## Best Practices 1. Set appropriate `temperature` values in the sandbox: 1. Lower (0.2) for factual responses 2. Higher (0.8) for creative responses 2. Enable `stream: true` for real-time responses 3. Use `rewrite_query` for optimized RAG queries 4. Include relevant file and URL content for context-aware responses # Knowledge base ## Overview Knowledge base is a central repository where you can upload and manage files that you want to embed and utilize within LLM Labs platform. It is designed to store documents that can be used for various purposes, such as enhancing understanding and leveraging them in the Sandbox for model development. ## Get started You can visit the **Knowledge base** page from the LLM Labs sidebar.
### Knowledge base creation 1. Click the **Create knowledge base** button. 2. Enter a knowledge base name, then click **Create**.
3. You can upload your desired files into the knowledge base by clicking the **Upload files** button. The maximum file size to be uploaded is 500MB.
{% hint style="info" %} You can also add files from external object storage. Learn more about adding files from [external object storage](/llm-projects/knowledge-base/external-object-storage). {% endhint %} 4. After you select the files, you will need to configure the global knowledge base configuration. You will only be asked about this once for each knowledge base, and it will be saved for future embeddings. You can change the settings later, either globally or individually.
The configurations are: * **Embedding model**: Your preferred embedding models. Datasaur supports several embedding models by default from these providers: * **OpenAI** * text-embedding-ada-002 * text-embedding-3-small * text-embedding-3-large * Text Embedding Ada 002 * Text Embedding 3 Small * Text Embedding 3 Large * **Amazon Bedrock** * amazon.titan-embed-text-v1 * amazon.titan-embed-image-v1 * amazon.titan-embed-text-v2:0 * cohere.embed-english-v3 * cohere.embed-multilingual-v3 * **Vertex AI** * textembedding-gecko\@003 * text-embedding-004 * textembedding-gecko-multilingual\@001 * text-multilingual-embedding-002 * **Chunk size**: The maximum number of characters that a chunk can contain. The larger the numbers, the bigger each chunk will be, allowing more data to be included within it. * **Overlap**: The number of characters that should overlap between two adjacent chunks. The larger the overlap, the more information each chunk shares with its neighboring chunks. * **Advanced settings**: Additional settings can enhance your data organization by enabling you to provide information about the file using the [file properties](/llm-projects/knowledge-base/file-properties) feature. 5. You can either process the files immediately by clicking **Process files**, or preview the chunking results and make modifications by clicking **Preview and edit.** 6. After completing the embedding process, you can preview the files and use them to conduct Retrieval-Augmented Generation (RAG) in LLM Labs.
### Add URLs You can also add URLs to the knowledge base, expanding the sources of information beyond file uploads. 1. Open your knowledge base, click on the more menu in the **Upload files** button, and select **Add URLs**.
2. A dialog box will appear where you can type your URLs. Press `Enter` to add URLs to the list.
3. Once you're done, click **Update knowledge base**.
4. You can either process the URLs immediately by clicking **Process files**, or preview the chunking results and make modifications by clicking **Preview and edit.**
## Search The search function allows you to validate the effectiveness of your knowledge base in providing context. The search results are shown in chunks that follow the chunk size and overlap value you specified. Each chunk will have a similarity score along with its source. A higher similarity score means the chunk content is more related to the given query.
## Activity The **Activity** tab logs all actions performed on your knowledge base, making it easier to track changes and actions. You can filter activity by member, source, file, or date.
## RAG example: Healthcare assistant Here is how a knowledge base can streamline the development of a Retrieval-Augmented Generation (RAG) based healthcare assistant in LLM Labs: 1. Create a [Sandbox](/llm-projects/sandbox). 2. Navigate to a model, and select the knowledge base you've created.
3. Write a prompt related to the knowledge base's content, then click **Run selected**. The results will be generated using the knowledge base content.
4. Below the generated completion, you can view the corresponding chunks. These are the parts of your knowledge base used to create the response.
5. You can also view the source file for the corresponding chunks by clicking the source link.
# External Object Storage ## Overview External object storage integration improves knowledge base accessibility by allowing users to connect their repositories directly. This enables straightforward data import, supporting a smoother and more efficient RAG process within the platform. ## Available providers Right now, we support five external object storage services: 1. AWS S3. 2. Google Cloud Storage. 3. Azure Blob Storage. 4. Dropbox. 5. Google Drive. ## Connect external object storage to workspace 1. Go to **Workspace settings.** 2. Navigate to **External object storage** section, then click **Add external object storage**.

Add external object storage

3. Choose a service and fill in the credentials.

List of object storage

For more detailed guides on connecting to each external object storage, you can refer to this documentation: 1. [AWS S3](/integrations/external-object-storage/aws-s3) 2. [Google Cloud Storage](/integrations/external-object-storage/google-cloud-storage) 3. [Azure Blob Storage](/integrations/external-object-storage/azure-blob-storage) 4. [Dropbox](/integrations/external-object-storage/dropbox) 5. [Google Drive](/integrations/external-object-storage/google-drive) ## Connect external object storage to knowledge base 1. Make sure you already connect the external object storage from the workspace settings. 2. Open your knowledge base, click the more menu next to **Upload files** button and select **Connect object storage.**
3. Select the object storage.
4. After you click your desired external object storage, a dialog will show. By default, all supported files will be added to the knowledge base.
5. You can filter the files by writing [rules](#rules-in-external-object-storage), then click **Test rules.** When the document list already matches your preference, click **Connect object storage.**
6. Once the process is finished, you can view your files within your knowledge base.
*** ### Rules in external object storage Rules help import specific files from external object storage into the knowledge base. Using [glob patterns](https://en.wikipedia.org/wiki/Glob_\(programming\)), you can include or exclude specific files.
When setting up rules for importing files from your external object storage into the knowledge base, understanding how to effectively use Glob patterns is crucial. Here are some tips to guide you through the process: #### Identify file patterns Determine the common patterns in the names of files you wish to import or exclude. For example, if you want to import all `.pdf` files, your pattern would be `*.pdf`. #### Utilize wildcards * `*` (asterisk) matches zero or more characters. For instance, `*.pdf` matches all files ending in `.pdf`. * `?` (question mark) matches exactly one character. For example, `?.pdf` matches `a.pdf` but not `ab.pdf`. #### Specify directories If you want to specify files in a particular directory, include the directory name in your pattern. For example, `myfolder/*.pdf` matches all `.pdf` files in the `myfolder` directory. #### Exclude files To exclude files, you can use the negation pattern `!`. For example, if you want to import all `.pdf` files except those starting with `temp`, your rules would include `*.pdf` and `!temp*.pdf`. #### Combine patterns You can combine multiple patterns to fine-tune your selection. For example, to import `.pdf` and `.docx` files but exclude those in the `drafts` folder, use `*.pdf`, `*.docs`, and `!drafts/*`. #### Example rules 1. Import all `.pdf` files: `*.pdf` 2. Import all files except those in the `temp` directory: `*`, `!temp/*` 3. Import all `.docx` and `.txt` files in the `data` directory: `data/*.docx`, `data/*.txt` Remember, the order of rules matters. Patterns defined later can override those defined earlier, so plan your rules accordingly to ensure the correct files are imported. # File Properties File properties act as metadata for your files. They let you add details such as type, category, or other attributes that describe the file. This metadata helps organize your knowledge base and makes the retrieval process more accurate and efficient in the future. ## Create file properties In a knowledge base, you can create file properties from: * **Knowledge base configuration dialog:** Open the **Advanced settings** accordion and create them there.
* **File properties section:** Open this section in the right sidebar and create them there.
There are two ways for you to create file properties: * **Upload a file properties file:** You can drag and drop the file into the dropdown area. Our supported formats are as follows: `.csv` and `.json`. For `.json` files, please find below a sample of our accepted format. {% file src="/files/BGKrh4bgP8vFY9jhbwQR" %} * **Create from scratch:** You can manually add a list of file property fields directly in the knowledge base.
## File property types Let's take a look at the available types of file properties. ### Text field **Text field** lets you give information by typing in a short free-form text, up to a single line at a time.
During configuration, you can also configure these settings: * **Required:** Specify whether this property must be filled. * **Allow multiple answer:** Specify whether users can provide more than one answer. * **Validation (in regex):** Specify patterns to ensure the input matches a specific format (example: email, phone number).
### Text area **Text area** lets you enter multiple lines of text, making it suitable for longer responses or detailed information.
During configuration, you can also configure these settings: * **Required:** Specify whether this property must be filled. * **Allow multiple answer:** Specify whether users can provide more than one answer.
### Dropdown **Dropdown** lets you select an option from a list. The list is hidden until they expand it.
During configuration, you can configure these settings: * **Dropdown options:** Add a list of options. If you already have a `.csv` file with a predefined options, upload it as an option set. * **Required:** Specify whether this property must be filled. * **Allow multiple answer:** Specify whether users can provide more than one answer.
* You can also allow selecting multiple answers by enabling **Allow multiple answers**. ### Hierarchical Dropdown **Hierarchical dropdown** lets you select options from categories and subcategories. The list stays hidden until expanded.
During configuration, you can configure these settings: * **Dropdown options:** Add a list of options. If you already have a `.csv` file with a predefined options, upload it as an option set. You can download the sample here: [Sample option set (Hierarchical)](https://drive.google.com/file/d/1ZsSTZ138Ld2Kkcs_BHPLksP4StPrDbLY/view?usp=sharing). {% hint style="info" %} **Option set file components** **1. The header** `id,label` is the header. This will always be the first row in the .csv. The first label will have `1` as the id. **2. id format** The id format is similar to Microsoft Word's numbering format. In the example above, `Characters` is a part of `Harry potter` and the id will be 1.1. 1. `Harry potter`: the root-level. 2. `1`: id for the root-level 3. `Characters`: the second-level. 4. `1.1`: the second-level id. {% endhint %} * **Required:** Specify whether this property must be filled. * **Allow multiple answer:** Specify whether users can provide more than one answer.
**Date** lets you select a specific date using the calendar picker or enter it manually in the required format.
During configuration, you can configure these settings: * **Required:** Specify whether this property must be filled. * **Date format:** Choose the format in which the date should be displayed (example: `DD-MM-YYYY`). * **Date separator:** Select the character that separates day, month, and year ( `-` or `/`). * **Use current date as default value:** Enable this option to automatically fill the field with the current date.
### Time **Time** lets you select a specific time using the clock interface or enter it manually in the required format.
During configuration, you can configure these settings: * **Required:** Specify whether this property must be filled. * **Time format:** Choose the format in which the time should be displayed (example: `HH:mm`). * **Time separator:** Select the character that separates hours and minutes ( `:` or `.`). * **Use current time as default value:** Enable this option to automatically fill the field with the current time.
### Slider **Slider** lets you adjust a value by sliding a handle along a bar, often used for selecting a range or level, such as volume.
During configuration, you can configure these settings: * **Required**: Specify whether this property must be filled. * **Slider range**: Define the minimum and maximum values for the slider. * **Hide slider value from labelers**: Choose this option to conceal the slider value from labelers. * **Slider range color** * **Start at**: Select the initial color. * **End at**: Select the final color. * You can also preview your slider configuration.
**Checkbox** is a simple box that you can check or uncheck to indicate a true or false decision.
During configuration, you can configure these settings: * **Required**: Specify whether this property must be filled. * **Hint**: Provide an optional hint to guide users in their input.
### URL **URL** is a field specifically meant for entering web addresses.
During configuration, you can configure these settings: * **Required:** Specify whether this property must be filled. * **Allow multiple answer:** Specify whether users can provide more than one answer.
### Radio button **Radio button** is set of options that are immediately visible, where you can select only one choice at a time.
During configuration, you can adjust the following settings: * **Required**: Specify whether this property must be filled. * **Hint**: Provide an optional hint to guide users in their input. * **Radio button options**: Enter the choices available for selection. The maximum number of options is limited to 25.
### Grouped attributes **Grouped attributes** lets you combine multiple properties that pertain to a single group.
During configuration, you can include any of the property types listed above in grouped attributes.
## Save file properties After the file properties are configured, they appear on the right side of each file. From there, you can add additional information for each file for better retrieval.
# Chunk Editor ## Overview Chunk editor in knowledge base provides you with granular control over each chunk of text extracted from your files. You can modify, split, insert, or delete chunks directly, helping you refine the information used in your Retrieval-Augmented Generation (RAG) systems. This leads to responses that are more accurate, relevant, and context-aware. ## Get started 1. Navigate to the **Knowledge base** page and open an existing one. 2. Expand the **Chunks** section from the sidebar on the right.
3. Once the **Chunks** section is opened, you can do the following for each chunk: 1. **Split chunk**: Divide the current chunk into two at the cursor position. 2. **Edit chunk**: Modify the content of the current chunk. 3. **Insert chunk above**: Add a new chunk above the current one. 4. **Insert chunk below**: Add a new chunk below the current one. 5. **Delete chunk**: Delete the current chunk. By deleting the chunk, that part will no longer be used for RAG.
{% hint style="info" %} You can also **double-click** a chunk to make changes, or **right-click** to view available actions and options. {% endhint %} 6. You can also see the metadata of the chunk by opening the **Metadata** section.
# Periodic Sync ## Overview Your knowledge bases can now stay in sync with your external object storage (AWS S3, Google Cloud Storage, Azure Blob Storage) through the periodic sync feature. It ensures your knowledge base is always updated without manual effort. By default, knowledge bases will **sync every hour** to import new or updated documents and remove outdated ones according to your rules. ## Configuration 1. Navigate to the **Sync schedule** section during external object storage setup.
2. Choose **Custom** to customize automatic synchronization. 3. Choose **Disable automated sync** if you prefer only manual updates. 4. Click **Save and update** once you have configured the schedule. ## Monitor synchronization status and schedule For each external object storage, a **sync status** icon is shown. Hover over the icon to see the last sync date.
The popover provides the following information: * **Last sync:** The time of the most recent successful sync. * **Next Sync:** The scheduled time of the next automatic sync. You can also do the following: * **Sync now:** Start an immediate, on-demand sync. It's useful for pulling in critical updates outside the regular schedule. * **Edit schedule:** Open the settings to change how often automatic sync runs. # Models ## Overview The **Models catalog** page allows you to explore and deploy LLMs. It includes over 200 base models, so you can pick one that fits your needs. You can connect your own models by adding credentials from our supported providers, or try **Direct Access LLMs** that consist of popular models you can use right away in Datasaur without entering credentials. ### Explore the models 1. Go to the **Models catalog** page under LLM Labs menu.
2. Navigate to the **Explore** tab.
### Integrate LLM providers You can also integrate with several LLM providers, such as Amazon SageMaker JumpStart, Amazon Bedrock, Azure OpenAI, OpenAI, and Google Vertex AI, through the **Manage providers** button.

Manage providers dialog

To connect a provider, you need to set up and add your credentials in Datasaur.

Input the credentials

Once you connect a provider, some will fetch your models so you can use them right away, while others require you to deploy the model first.
### Deploy the models Here’s how to deploy a model from Amazon SageMaker JumpStart. Click **Deploy model**, then in the dialog that appears, enter an endpoint name and choose your preferred instance type.

Deploy model dialog

After you click **Deploy model**, the deployment may take several minutes. You can track its progress in the model status at the top right.

Deploying process

After deployment is complete, the model status will change to **Available**, and you can start using the model right away.

Deployed model

## Available models The **Available** tab shows all deployed models and lets you deploy or undeploy them. It also lists our **Direct Access LLMs**, popular models you can use right away without having to add credentials.
{% hint style="info" %} Every new model that you deploy in the LLM providers will be synced to Datasaur, and you can use it right away in Datasaur. If the models you just deployed haven't appeared on **Available** tab, you can click the **Sync models** button. {% endhint %} ### Undeploy models To undeploy a model, you can click the three dots on the model card and select **Undeploy model**.

Undeploy models

The model status will be changed into unavailable once you click the **Undeploy model** button.

Unavailable models

### Disconnect LLM providers To disconnect a provider, click **Manage providers**, then **View details** for the provider you want to disconnect. In the dialog that opens, click **Disconnect** at the bottom left.

Disconnect dialog

Once you’ve clicked the **Disconnect** button, the providers will be disconnected from your workspace. # Amazon SageMaker JumpStart ## Overview Amazon SageMaker JumpStart integration in LLM Labs lets you connect and manage your own models and deployed new ones in Datasaur, making them available for evaluation and deployment within LLM Labs. ## Setup ### Connect to Amazon SageMaker Jumpstart 1. Select **Amazon SageMaker JumpStart.**
2. An External ID is generated each time the dialog is opened. This ID will be used later when creating the role ARN. To avoid regenerating a new External ID, **do not close the dialog during setup**. Otherwise, you'll need to create the Role ARN with a different External ID. 3. You can start by selecting the **Region** attribute. We'll get back to this window later. Let's leave it for now.
### Create a role ARN Create a role which will be assumed by Datasaur to access your Amazon SageMaker Jumpstart. 1. Open the [IAM console](https://console.aws.amazon.com/iam/). 2. Go to **Roles** and then click **Create role**. 3. Choose **Custom trust policy** in the trusted entity type section.

Custom Trust Policy

4. Paste the following configurations in custom trust policy. Don't forget to replace the **ExternalID.** After that, click **Next**. ``` { "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Principal": { "AWS": "682361690817" }, "Action": "sts:AssumeRole", "Condition": { "StringEquals": { "sts:ExternalId": "" } } }, { "Effect": "Allow", "Principal": { "Service": "sagemaker.amazonaws.com" }, "Action": "sts:AssumeRole" } ] } ``` 5. In the **Add permissions** section, select the **AmazonSageMakerFullAccess** policy. Then, click **Next**.
6. Enter a **Role name** and a **Description** (optional). 7. Click **Create role**. 8. After that, back on the **Roles** page, click your newly created role. 9. Copy the **Role ARN** from the page and paste it in the **Amazon SageMaker Jumpstart** dialog in Datasaur. Then, click **Add Credentials** to connect. ## **Disconnect** 1. Click **Manage providers**, then click **View details** on Amazon SageMaker JumpStart provider.
2. In the dialog that opens, click **Disconnect** at the bottom left.
3. All Amazon SageMaker JumpStart models linked to that credentials will be removed from the workspace. # Amazon Bedrock ## Overview Amazon Bedrock integration in LLM Labs lets you connect your own models to Datasaur, making them available for evaluation and deployment within LLM Labs. It enhances security compliance by giving you complete control and privacy over your data. By using your own LLMs, you can ensure that sensitive information remains within your organization's ecosystem. ## Setup ### Connect to Amazon Bedrock 1. Select **Amazon Bedrock.**
2. Enter your external ID, Role ARN, and Region. Learn more on [how to create a role ARN](#create-a-role-arn-for-amazon-bedrock).
3. Once you have connected Amazon Bedrock to LLM Labs, you will see a list of available LLMs that you have already deployed. You can immediately use these models within LLM Labs.
{% hint style="info" %} Every new model that you deploy in Amazon Bedrock will be synced to Datasaur, and you can use it right away in Datasaur. If the models you just deployed haven't appeared on Datasaur, you can click **Sync models**. {% endhint %} ### Create a Role ARN for Amazon Bedrock 1. Open the [IAM console](https://console.aws.amazon.com/iam/). 2. Go to **Roles** and click **Create role**. 3. Choose **Custom trust policy** in the trusted entity type section.
4. Paste the following configurations in custom trust policy. Don't forget to replace the **ExternalID**. After that, click **Next**. ```json { "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Principal": { "AWS": "682361690817" }, "Action": "sts:AssumeRole", "Condition": { "StringEquals": { "sts:ExternalId": "" } } }, { "Effect": "Allow", "Principal": { "Service": "bedrock.amazonaws.com" }, "Action": "sts:AssumeRole" } ] } ``` 5. In the **Add permissions** section, select the **AmazonBedrockFullAccess** policy. After that, click **Next**.
6. Enter a **Role name** and a **Description** (optional). 7. Click **Create role**. 8. After that, go back to the **Roles** page, and click on your newly created role. 9. Copy the **Role ARN** from the page and paste it in the **Amazon Bedrock** dialog in Datasaur. Then, click **Add Credentials** to connect. ## Supported Amazon Bedrock models Datasaur supports a wide range of models available in Amazon Bedrock. You can find the list of supported models here: [Amazon Bedrock Models](https://docs.aws.amazon.com/bedrock/latest/userguide/models-supported.html). {% hint style="info" %} Currently, Datasaur only supports text generation LLMs. {% endhint %} Additionally, you can bring your own **fine-tuned models** from Amazon Bedrock into Datasaur, allowing for greater customization and performance tailored to your specific use cases. Learn more about [fine-tuning](/llm-projects/models/fine-tuning). ## Disconnect Amazon Bedrock To disconnect Amazon Bedrock from Datasaur, follow these steps: 1. Click **Manage providers**, then click **View details** on Amazon Bedrock provider.
2. In the dialog that opens, click **Disconnect** at the bottom left.
## Availability The models will be accessible to all workspace members for use in their projects. Additionally, only the **Admin** can remove the Amazon Bedrock provider from the workspace. # Open AI OpenAI integration in LLM Labs lets you connect and manage OpenAI chat-completion models, making them available for evaluation and deployment in LLM Labs. ## Setup 1. Click **Manage providers** then select **OpenAI** in the dialog that appears.
2. Enter your API key. [Learn more about how to find your OpenAI API key](https://platform.openai.com/api-keys)
3. Click **Add Credentials**. After the connection is successful, all models available through your API key will be synced to Datasaur. ## Disconnect 1. Click **Manage providers**, then click **View details** on OpenAI provider.
2. In the dialog that opens, click **Disconnect** at the bottom left.
3. All OpenAI models linked to that API key will be removed from the workspace. # Azure OpenAI ## Overview Azure OpenAI integration allows you to integrate your own models into Datasaur. This functionality provides three key benefits: 1. **Integrating your own LLMs**: This feature enables you to bring your own models into LLM Labs. Datasaur will retrieve all available Azure OpenAI models based on your credentials. Once connected, these models will be accessible to all workspace members. 2. **Protect your data**: By using your own LLMs, you can ensure better security compliance, providing you with complete control and privacy over your data. 3. **Streamlined workflow:** Access your preferred models directly within Datasaur's user-friendly interface, simplifying your LLM workflows to allows you to gain insights about your model's performance by comparing it to other models, both within and outside of Azure. ## Setup ### Integrate Azure OpenAI using application role 1. Navigate to the **Models** page in LLM Labs. 2. Open **My models** tab, and click the **Manage providers** button. 3. Choose **Azure OpenAI** as the provider, and enter your tenant ID, subscription ID, resource group name, and account name.
4. If this is your first time integrating Azure Open AI with Datasaur, click **Add Datasaur to your Azure Tenant** to grant consent for Datasaur's App Registration to your tenant. If you have previously done this, you can skip this step. * You must have at least the **Privileged Role Administrator** role to grant consent to **App Registrations** for delegated access. Continue the process on a new tab and authorize the Datasaur app. Note that you need **Admin** permission to complete this action.
5. Grant role assignments to Datasaur's Service Principal 1. Still in the **Azure OpenAI Account**, find **Access Control (IAM)** on the sidebar. 2. Click **Add** **>** **Add role assignment**. 3. Select **Cognitive Services OpenAI Contributor**, then click **Next**. 4. In **Assign access to**, click **User, group, or service principal**, then click **select members**. 5. Search for **Datasaur Azure AI Integrator** (you cannot see the service principals from the list, you must search for them), then add it. This step will not be available if you haven't added Datasaur to your Azure tenant as described above. 6. Click **Next**, and when you reach the **Conditions** section, click **Next** again. 7. Click **Review + assign**. 8. Get the tenant ID: 1. Go to your **Microsoft** **Entra ID** from the Azure portal. [Learn more about Microsoft Entra ID](https://learn.microsoft.com/en-us/entra/fundamentals/whatis). 2. Under the **Overview** tab, copy the **Tenant ID** attribute under the **Basic information** section. 9. Get the subscription ID, resource group name, and account name: 1. Go to the **Azure OpenAI** page from the Azure portal. 2. Create or select an **Azure OpenAI Account**. 3. From the sidebar, click **Overview**. 4. You should find these fields under the **Essentials** section. **Subscription ID** should be available as **Subscription ID, Resource group name** should be available as **Resource group**, and **Account name** is the name of the selected **Azure OpenAI account.**
10. Once you have connected your Azure OpenAI Models to LLM Labs, you will see a list of available LLM Models that you have already deployed in Azure OpenAI. You can immediately use these models within LLM Labs.
{% hint style="info" %} Every new model that you deploy in Azure OpenAI will be synced to Datasaur, and you can use it right away in Datasaur. If the models you just deployed haven't appeared on Datasaur, you can click **Sync models**. {% endhint %} ### Integrate Azure OpenAI using application certificate This section provides a step-by-step guide for integrating Azure OpenAI with LLM Labs using a certificate for secure authentication. You’ll learn how to generate credentials, upload the certificate to Azure, and connect Azure OpenAI to LLM Labs. {% hint style="info" %} If you are connecting to Azure OpenAI using a certificate, note that only the deployment name that matches the format `modelName-modelVersion` will be integrated with LLM Labs. {% endhint %} #### Step 1: Generate credentials To integrate Azure Open AI using a certificate, you need to generate a private key, extract the public key, and combine them into a single PEM file. 1. Generate a private key. If you don’t have a private key, generate one using the following command: `openssl genrsa -out private.pem 2048` 2. Extract the public key from the private key. Use the private key to generate a public key in `.pem` format: `openssl req -new -x509 -key private.pem -out public.pem -days 365` 3. Combine keys into one `.pem` file: `cat private.pem public.pem > fullkeys.pem` #### Step 2: Upload the certificate to Azure App Registration 1. Access **Azure Portal:** Log in to the [Azure Portal](https://portal.azure.com/). 2. Go to **Azure Active Directory** **>** **App registrations**. 3. Select your application. 4. Upload the certificate: 1. In the left menu, go to **Certificates & secrets**. 2. Under the **Certificates** section, click **Upload certificate**. 3. Choose your `.pem` (public key) file and upload it. 5. Retrieve tenant ID and client ID: 1. On your application page, go to the **Overview** section. 2. Copy the **Tenant ID** and **Client ID**. These will be required to connect Azure Open AI to LLM Labs. 3. Ensure your application has access to the Azure Open AI service you want to connect to. #### Step 3: Connect Azure Open AI Provider in LLM Labs 1. Navigate to the **Models catalog** in your Datasaur workspace, and click **Manage providers**. 2. Choose **Azure OpenAI** as the provider, and select **Azure tenant application certificate**.
3. Enter your **Tenant ID**, **Client ID**, and **Azure OpenAI** **endpoint**. 4. Upload the `.pem` file containing both the private and public keys (created in step 1). ## Supported Azure OpenAI models Datasaur supports a wide range of models available in Azure OpenAI. You can find the list of supported models here: [Azure OpenAI Models](https://learn.microsoft.com/en-us/azure/ai-services/openai/concepts/models). {% hint style="info" %} Currently, Datasaur only provides support for text generation LLMs. {% endhint %} Additionally, you can bring your own **fine-tuned models** from Azure OpenAI into Datasaur, allowing for greater customization and performance tailored to your specific use cases. ## Disconnect Azure OpenAI To disconnect Azure OpenAI from Datasaur, follow these steps: 1. Click **Manage providers**, then click **View details** on Azure OpenAI provider.
2. In the dialog that opens, click **Disconnect** at the bottom left.
## Availability The model will be accessible to all workspace members for use in their projects. Additionally, only the **Admin** can remove the Azure OpenAI provider from the workspace. # Vertex AI Vertex AI integration in LLM Labs lets you connect and manage your Vertex AI models, making them available for exploration, evaluation, and deployment within LLM Labs. ## Setup 1. Click **Manage providers** then select **Google Vertex AI** in the dialog that appears.
2. Insert your client email, private key, project ID, and region. To access the region: 1. From **Google Cloud Console**: 1. Go to **Vertex AI** section. 2. Look at the region selector in the top navigation bar, or check where your Vertex AI resources are deployed. 2. From **Project Settings**: 1. Go to **IAM & Admin > Settings**. 2. Look for **Location** or **Region**.
{% hint style="info" %} Use the correct region format as specified in [Vertex AI locations](https://cloud.google.com/vertex-ai/docs/general/locations), for example, `us-east5`. {% endhint %} 3. Once you have connected Vertex AI to LLM Labs, you will see a list of available LLMs that you have already deployed. You can immediately use these models within LLM Labs.
{% hint style="info" %} Every new model that you deploy in Vertex AI will be synced to Datasaur, and you can use it right away in Datasaur. If the models you just deployed haven't appeared on Datasaur, you can click **Sync models**. {% endhint %} ## Availability The models will be accessible to all workspace members for use in their projects. Only the **Admin** can remove the Vertex AI provider from the workspace. ## Disconnect 1. Click **Manage providers**, then click **View details** on Google Vertex AI provider.
2. In the dialog that opens, click **Disconnect** at the bottom left.
3. All Vertex AI models linked to that credentials will be removed from the workspace. # Custom model Integrate your own LLMs to Datasaur with custom models, and use them for exploration in Sandbox or for evaluation. ## Integration While Datasaur offers direct integrations with some providers, you can also connect to models hosted on various third-party platforms or your own infrastructure using the **Custom model** feature. {% hint style="info" %} To simplify integration, Datasaur's custom model connection adheres to the widely adopted API structure defined by OpenAI for its completions or chat completions endpoints. This means that if your self-hosted model or third-party serving framework exposes an endpoint that mimics the OpenAI API format, connecting it to Datasaur is straightforward. {% endhint %} ### API specification To simplify integration, Datasaur connects to custom models through an API structure compatible with the OpenAI Chat Completions API. Think of it as using a standard interface: if your model serving framework (such as TGI) exposes an OpenAI-compatible API, Datasaur can connect to it directly with minimal configuration. ### How the API connection works {% hint style="info" %} Please note that the `streaming` option is currently disabled for custom models. {% endhint %} When Datasaur uses your custom model, here’s what happens: #### 1. Datasaur sends a request Datasaur sends information to your model's address using a standard web method (`POST`). This request goes to a specific path, usually `/v1/chat/completions`, added to the main address you provide. The request includes: * **Headers:** Tells the server the data is in JSON format. If you added an API Key in Datasaur, it sends that key for security (`Authorization: Bearer YOUR_API_KEY`). * **JSON body:** The request body contains: * `model`: The name of the specific model you want to use. * `messages`: The conversation history, including instructions ("system" message) and the user's input ("user" message). * Optional settings like `temperature` (for creativity) or `max_tokens` (to limit response length). Example request data: ```json { "model": "tgi", "messages": [ { "role": "system", "content": "You are a helpful assistant." }, { "role": "user", "content": "Explain what an LLM is in simple terms." } ], "temperature": 0.7, "max_tokens": 150 } ``` #### 2. Your model sends a response If the request succeeds, your model server returns a `200 OK` response with a JSON payload. The response typically includes: * `id`: A unique ID for this conversation turn. * `model`: The model used to generate the response. * `choices`: An array (usually just one item) containing the model's reply: * `message`: The actual text generated by the model (`content`) and its role (`assistant`). * `usage` (Optional): Token usage statistics. Example response data: {% code overflow="wrap" %} ```json { "id": "chatcmpl-randomid12345", "object": "chat.completion", "created": 1713867000, // Timestamp "model": "your-model-name", "choices": [ { "index": 0, "message": { "role": "assistant", "content": "An LLM, or Large Language Model, is like a super-smart computer program that's read tons of text, so it can understand and write text almost like a human!" }, "finish_reason": "stop" // Why the model stopped writing } ], "usage": { "prompt_tokens": 30, "completion_tokens": 55, "total_tokens": 85 } } ``` {% endcode %} ### Text Generation Inference (TGI) & Hugging Face * **Main Address (Base URL):** Enter the address where your TGI server is running. This might look like `http://your-tgi-server-ip:8080/v1` or maybe use port `80` if using certain hosting like Hugging Face Inference Endpoints. Add `/v1` at the end. * **Model Name (in request data):** TGI usually runs one main model at a time. You might just need to put `"tgi"` as the model name, or use the specific Hugging Face name the model was started with (like `"NousResearch/Nous-Hermes-2"`). Check your TGI setup. [Learn more about TGI](https://huggingface.co/docs/text-generation-inference/en/basic_tutorials/consuming_tgi#curl). * **API Key:** If you're using TGI, especially through services like Hugging Face, you'll likely need an API key. Get this key from your TGI provider (like your Hugging Face Access Token), and put it in the API Key field in Datasaur's custom model settings. ## Connect custom models To connect a custom model: 1. Navigate to the **Models catalog** page, then click **Manage providers**. 2. Select **Custom model**.
3. Input your model credentials. The required credentials are: 1. **Endpoint URL**: The endpoint URL of your model. 2. **API key**: The API key of your model. 3. **Model name**: Your desired model name to be used in LLM Labs.
{% hint style="info" %} If you are adding a custom model from an LLM provider like Hugging Face, you only need to input the endpoint URL without the `/v1/chat/completions` suffix. {% endhint %} 5. Once you’ve added your credentials, click the **Add custom model** button, and your custom model will be available in LLM Labs.

Models available

## Manage custom models To manage your custom models, click the three-dot menu on the model card. From there, you can: * View details. * Try in Sandbox. * Edit. * Delete.
### Try in Sandbox Click **Try in Sandbox,** and you'll be automatically taken to a Sandbox. This allows you to use it as a base model and test how it works with additional instructions and various prompts. Learn more about [Sandbox](/llm-projects/sandbox).

Try in Sandbox

### Edit custom model Click **Edit** to modify the endpoint URL, API key, and model name. Once you've updated the model credentials, click **Save custom model** to save your changes.
### Delete custom model Click **Delete** to delete your model from LLM Labs. Confirm the deletion by clicking the **Delete custom model** button.

Delete custom model

# Fine-tuning ## Overview LLMs in general are typically trained for generic use cases. Fine-tuning LLMs allows users to further train the model to provide more accurate answers for a specific domain or use case. This process involves providing the model with a dataset containing examples of input and output from a specific domain. LLM Labs helps simplify this process by providing a user-friendly way to fine-tune and deploy open-source models, allowing you to tailor LLMs to your exact needs. ## Fine-Tuning Providers There are two main options available for fine-tuning: 1. **Served by Datasaur** In this option, the fine-tuning process is fully managed and hosted by Datasaur. You don’t need to set up any infrastructure or manage cloud resources — Datasaur will handle everything for you. Currently, we support the following providers and pre-trained model(s) for fine-tuning served by Datasaur: 1. **Amazon Bedrock** 1. Amazon Nova Micro 2. Amazon Nova Lite 3. Amazon Nova Pro 4. Amazon Titan Text G1 - Express 5. Amazon Titan Text G1 - Lite 6. Cohere Command 7. Cohere Command Light 8. Meta Llama 3.1 8B 9. Meta Llama 3.1 70B 2. **Bring Your Own Key (BYOK)** In this option, the fine-tuning process runs in your own cloud account. This gives you full control over the environment — you can configure machine types, regions, and other resources as needed. BYOK is ideal for teams that require advanced customization. Currently, we support fine-tuning with BYOK through [Amazon SageMaker Jumpstart](https://docs.datasaur.ai/llm-projects/models/amazon-sagemaker-jumpstart) for various models from **Meta Llama**\*, **AI Singapore**, **Qwen**, and **DeepSeek**. Only **instruct models** are supported for fine-tuning at this time. If you’d like to fine-tune a model not yet available in LLM Labs, please contact our support team at .\ BYOK models requires additional prerequisites before you are able to fine-tune: * Your customer plan type should be **Subscription**, **Enterprise Usage Credit**, or **Internal** * Connect to the selected provider through [Models catalog > Manage providers](https://docs.datasaur.ai/llm-projects/models/amazon-sagemaker-jumpstart). * Ensure your account role of the selected provider have necessary permission to perform fine-tuning. {% hint style="info" %} \*) Under the hood, **AWS SageMaker fine-tuning** uses the **Hugging Face** framework. Some base models, such as **Meta Llama**, are *gated models* — meaning you must log in to your Hugging Face account and accept the model’s license terms before using them. To fine-tune a gated model, you must store your **Hugging Face access token (`hf_token`)** as a **secret key** in your AWS account. Ensure that your AWS account has the necessary permissions to access this secret key. 🔒 **Datasaur does not have access to your secret key value** — it remains fully secured within your AWS environment. {% endhint %} ## Create fine-tuned models This section guides you through the process of fine-tuning your models in LLM Labs. ### Step 1: Set up model 1. Navigate to the **Models** page. 2. On the **Available** tab, go to the **Fine-tuned LLMs** section, and click **Create fine-tuned model**.

My models

3. Set up your fine-tuning job.
1. **Name your fine-tuned model**. This name uniquely identifies your fine-tuned model. 1. For AWS SageMaker, it will be used as the trainingJobId and must comply with the [AWS SageMaker’s naming requirements](https://docs.aws.amazon.com/sagemaker/latest/APIReference/API_CreateTrainingJob.html#API_CreateTrainingJob_RequestSyntax): 1. **Uniqueness:** Must be unique within the same AWS Region and AWS account. 2. **Length:** 1–63 characters. 3. **Allowed characters:** Letters (`a–z`, `A–Z`), numbers (`0–9`), and hyphens (). 4. **Format rules:** 1. Must start and end with an alphanumeric character. 2. Cannot include spaces, underscores (`_`), or special characters other than hyphens. 2. **Select a base model that you want to fine-tune**. You can select either: 1. Pre-trained LLMs provided by Datasaur, or 2. Existing fine-tuned models 2. Choose a dataset. You can upload either a .csv consisting of 2 columns: `prompt`, `expected completion`, or you can choose an existing dataset from the library. For the validation dataset, you have 3 options: 1. **Split from selected dataset**: Datasaur will split the uploaded dataset and use it for validation data. You will need to configure the validation size using a percentage.

Split from selected dataset

2. **Use new dataset**: You will need to add a new dataset to use as validation.

Use new dataset

3. **None**: Choose this option if you don't want to add a validation dataset. {% hint style="danger" %} Important: Please note that validation data are required for several base models (e.g., Cohere Command, Cohere Command Light). On the other hand, some models may not support validation data at all. {% endhint %} {% hint style="info" %} [Learn more about Dataset](https://docs.datasaur.ai/llm-projects/dataset) {% endhint %} 4. **Configure your resources**. This step applies to SageMaker only and determines the cost and performance of your training job. 1. **Instance type**: The machine used to fine-tune your model. 2. **Instance count**: The number of instances for training. 3. **Training volume size**: The storage capacity allocated to each instance. For detailed recommendations, see the [Choosing the Right Resources](#choosing-the-right-resources) section. ### Step 2: Adjust hyperparameters Next, you will need to configure the hyperparameters for your fine-tuning project.

Adjust hyperparameters

The fundamental hyperparameters are **epochs** and **learning rate**. Some models also support **advanced hyperparameters**, which Datasaur provides with default values. These parameters are always applied alongside the fundamental ones, but you can adjust them for further fine-tuning if desired. The advanced hyperparameters include: **early stopping callback** (for fine-tuning with validation), **batch size**, and **learning warm-up steps**. #### **Optional Hyperparameters** {% hint style="info" %} This section applies to **AWS SageMaker JumpStart Bring Your Own Key (BYOK)** fine-tuning only. {% endhint %}
If you wish to include additional hyperparameters, you can define them in **Optional Hyperparameters** using a **JSON schema**. For example, to enable **LoRA**, you can specify: ```json { "use_lora": true, "lora_config": { "r": 8, "lora_alpha": 16, "lora_dropout": 0.05 } } ``` You **cannot override** the fundamental or advanced hyperparameters in **Optional Hyperparameters**. If you partially specify hyperparameters without providing its full configuration, the **default values** will be applied automatically. For example, if you configure `use_lora` without specifying `lora_config`, the default LoRA configuration will be: ```json { "lora_config": { "r": 8, "lora_alpha": 8, "lora_dropout": 0.0 } } ``` For more recommendation about hyperparameters, see the [Common Fine-Tuning Hyperparameters](#common-fine-tuning-hyperparameters) section. ### Step 3: Review job 1. The last step is to review your fine-tuning job before you start the process.

Review job

2. You can review the optional hyperparameters that you add by clicking **View optional hyperparameters**.

Review job

3. You can also view the predicted cost by clicking the **View total predicted cost** button on the **Costs** section. It will calculate and show you the total predicted cost for starting the fine-tuning process.

Review job

{% hint style="info" %} Please note that this is just a cost prediction. The final cost may be higher or lower, as each model has its own tokenizer. {% endhint %} 4. Once you have reviewed the configuration, you will need to check the acknowledgement checkbox. 5. Lastly, click **Start fine-tuning job** and the training process will start.

Training process

{% hint style="info" %} It will take several hours for the training process to be completed. Datasaur will notify you by email when the training process is complete. {% endhint %} 5. Once the training process is complete, your model will be available to deploy.

My models page

## Model management ### Model status There are 7 possible statuses for the fine-tuned models. 1. **Training**: The model is currently being trained on your dataset. This status indicates that the training process is in progress, and the model is learning from your data.

Training status

2. **Training failed**: The model training process has failed due to an error. This status indicates that the training process was interrupted, and you may need to investigate and resolve the issue.

Training failed

3. **Stopping training**: The model training process is being stopped. This status indicates that someone has chosen to stop the training.

Stopping training

4. **Training stopped**: The model training process has been stopped. This status indicates that the training process has been successfully stopped, and you can’t continue the training once it stopped.

Training stopped

5. **Not deployed**: The model has been trained but has not yet been deployed for use. You can deploy the model to use it in Sandbox.

Not deployed

6. **Deploying**: The model is being deployed for use. This status indicates that the deployment process is in progress, and the model will soon be available for use in Sandbox.

Deploying

7. **Deployed**: The model has been successfully deployed. This status indicates that the model is now available for use in Sandbox, and you can start using it to generate predictions or responses.

Deployed

### Deploy models To deploy a fine-tuned model: 1. Click **Deploy** **model** to start the deployment.
2. In the dialog that appears, specify the **auto undeploy schedule**.
3. Click **Deploy model** in the dialog to confirm and the process will start.

Deploying process

4. Once the process is finish, your model will be available to use for experiment in Sandbox. [Learn more about Sandbox](https://docs.datasaur.ai/llm-projects/sandbox)

Model deployed

### Undeploy models 1. Click the more menu (three-dots) in the right corner of the model card and select **Undeploy.**

Undeploy model

2. Confirm the process by clicking **Undeploy** in the dialog that appears.
3. Your model will be undeployed and you will no longer be charged for the hourly cost. ### View model details To view the model details, click the more menu (three-dots) and select **View details.** The details of the fine-tuned model will be shown.

View details menu

In this dialog, you can view the dataset, validation dataset, models used, hyperparameter configurations, the creator, and storage cost information.

View details menu

View details menu

### Download models {% hint style="info" %} This section applies to **AWS SageMaker JumpStart Bring Your Own Key (BYOK)** fine-tuning only. {% endhint %}

Download model

You can use your fine-tuned model for further tasks by **downloading it**. The model can then be loaded in your **local environment** for inference or additional fine-tuning, uploaded to your **own SageMaker or other cloud environment**, or backed up for **compliance or reproducibility purposes**. When you initiate a download, the selected fine-tuned model is **exported as a `.tar.gz` archive** that contains: * **Model weights:** The core parameters of the fine-tuned model. * **Configuration files:** Metadata needed to load the model correctly (e.g., tokenizer config, architecture). * **Optional adapters:** If the model includes LoRA or other PEFT adapters, these are included as separate files. ### Delete models To delete a fine-tuned model, click the more menu (three-dots) and select **Delete.**

Delete model

In the dialog that appears, check the acknowledgement checkbox, and confirm the deletion by clicking **Delete model.**
### Use in Sandbox Once a fine-tuned model is deployed, it will be available in the Sandbox for further experimentation and testing. This allows you to integrate and test the specialized model within your specific applications. [Learn more about Sandbox](https://docs.datasaur.ai/llm-projects/sandbox).

Access model via Sandbox

## Choosing the Right Resources {% hint style="info" %} This section applies to **Bring Your Own Key (BYOK)** fine-tuning only. {% endhint %} Every fine-tuning job has **different resource requirements** depending on **model size**, **dataset size**, and **training configuration** (e.g., batch size, sequence length, LoRA rank). Each instance differs in **GPU memory (VRAM)**, **number of GPUs**, and **storage capacity**, which affect training performance, duration, and cost. **Example:** * `ml.g5.4xlarge` — 1 GPU, 16 GB VRAM, 125 GB storage → not sufficient for 7B model * `ml.g5.12xlarge` — 4 GPUs, 64 GB VRAM (16 GB per GPU), 375 GB storage For detailed information, see the [Amazon SageMaker pricing documentation](https://aws.amazon.com/sagemaker/pricing/), specifically under the "Amazon Sagemaker Unified Studio Notebooks" section. Selecting the right instance ensures **stable training**, **optimal performance**, and **cost efficiency**. To do so, we first need to understand the fine-tuning requirements before choosing the most suitable instance for VRAM, storage, and instance count calculations. ### VRAM (GPU Memory) VRAM determines how large your model and training batch size can be. If VRAM is too small, your training may fail or become extremely slow due to gradient checkpointing or offloading. **Main factors affecting VRAM:** * **Model size (parameters)** — Larger models require more memory to store weights. * **Precision (FP32, FP16, BF16, FP8, etc.)** — Lower precision reduces VRAM usage, but might have tradeoffs with accuracy. * **Optimizer states** — Extra memory for momentum, variance, etc. (e.g., Adam/AdamW). * **Activations** — Temporary memory for forward/backward pass. * **Use of LoRA / PEFT** — Fine-tunes only a subset of weights, so it will reduce memory usage significantly. {% hint style="danger" %} Activations are not included in this simplified estimation, as they are usually small compared to weights, gradients, and optimizer states for typical batch sizes. {% endhint %} #### **Step 1: Identify model size** * Model has **7B parameters**. * This is the **baseline factor** for VRAM calculation. #### **Step 2: Apply precision** * Precision determines how much memory each parameter uses. * Common precisions: | Precision | Bytes per parameter | Memory effect | | -------------- | ------------------- | ---------------------------------- | | FP32 (default) | 4 | Full memory | | FP16 | 2 | Half memory of FP32 (×2 reduction) | | BF16 | 2 | Half memory of FP32 | * **Example:** * VRAM = 7B parameters × 2 bytes (FP16) = **14 GB** #### **Step 3: Add gradients** * Each weight requires a **gradient of the same precision**. * Memory for gradients is approximately the **same as weights**, so total VRAM doubles compared to Step 2. * **Example:** * VRAM = 14 GB (weights) + 14 GB (gradients) = **28 GB** #### **Step 4: Add optimizer states** * Optimizers like Adam/AdamW store **additional memory** for momentum and variance (\~2× weights). These extra tensors require additional memory for every trainable parameter. | Optimizer | Extra tensors | Approximate multiplier | | ------------ | ----------------------- | ------------------------------------------------ | | Adam / AdamW | 2 (momentum + variance) | 2× weights | | Adafactor | 1 (factored variance) | 1× weights (can be less in some implementations) | | SGD | 0–1 (momentum) | 0–1× weights | * **Example (AdamW):** * VRAM = 28 GB (weights + gradients) \* 28 GB (optimizer) = **56 GB** #### **Step 5: Adjust for training method (full vs LoRA)** * For full fine-tuning, all weights, gradients, and optimizer states are used. * **Example:** * VRAM = 56 GB (previous calculation) * LoRA fine-tunes only a **small fraction of model parameters**, instead of all of them. This reduces memory for **gradients + optimizer states** (approximately 10–20%), but **the full model weights still need to be loaded** for forward computations. * **Example** (Using 20% as fraction): * VRAM = Weights+(Gradients + Optimizer)×Fraction of trainable parameters * 14 GB + ((14 GB + 28 GB) \* 20%) = 14 GB + 8.4GB = 22.4GB Please note that these are **rough estimates**. Actual VRAM usage can vary depending on factors such as batch size, sequence length, precision, gradient checkpointing, and framework overhead, which can **increase memory requirements**. At the same time, certain techniques, like LoRA, reduce memory usage for gradients and optimizer states in proportion to the fraction of weights being trained, although the full model weights still need to be loaded. It is recommended to always leave s**ome extra GPU memory as a buffer** to account for unexpected spikes in memory usage, framework overhead, or temporary allocations during training, which helps prevent out-of-memory errors and ensures stable execution. ### Training Volume Size Storage determines how much space you have for the **model checkpoints**, **intermediate artifacts**, and **dataset**. #### **Step 1: Base model storage** * This is the size of the pre-trained weights. Most pre-trained models are stored in **FP16 or compressed format**. * **Example:** * 7B model (FP16) ≈ **14 GB** {% hint style="info" %} If stored in FP32, size roughly doubles (\~28 GB). {% endhint %} #### **Step 2: Checkpoints storage** * During training, **checkpoints are periodically saved** to allow recovery or evaluation. * Each checkpoint contains **model weights + optimizer states + metadata**. * Typical checkpoint storage ≈ **2–3× the base model size**, because: * Optimizer states (AdamW, Adafactor) require extra memory (\~1–2× weights) * Metadata and temporary artifacts add minor overhead * **Example (1x checkpoint):** * 7B model (FP16) → 14 GB × 2–3 ≈ **26–39 GB** {% hint style="info" %} More frequent checkpoints or larger optimizers increase storage requirements. {% endhint %} #### **Step 3: Dataset size** * Include the **raw dataset size** (CSV, JSON, etc.). * **Example:** 1 GB dataset adds 1 GB to volume #### **Step 4: Recommended volume size** * Storage Size ≈ Base model + Checkpoints + Dataset * **Example:** * 14 GB (weights) + 30 GB (checkpoints) + 1 GB (dataset) ≈ **35 GB** * **Recommendation:** Round up (e.g., 50 GB) to allow buffer for temporary files and unexpected artifacts. ### Instance Count Instance count determines **how many machines** are used for distributed training. * **Single-instance (1)** — Simpler setup, lower cost, but slower for large models. * **Multiple instances (≥2)** — Distributes model and data across machines for parallel training, but requires more setup time and inter-node communication. **When to increase instance count:** * Model is too large to fit in one GPU (e.g., 70B+) * Dataset is massive and you want to reduce training time {% hint style="danger" %} While multi-GPU is helpful to handle large model, beware of overhead (time, VRAM, and compute efficiency). {% endhint %} ## Common Fine-Tuning Hyperparameters Hyperparameters control how the model learns and affect **training stability**, **performance**, and **final model quality**. Choosing appropriate values balances **training efficiency**, **accuracy**, and **resource cost**. Common fine-tuning hyperparameters include: ### **Fundamental Hyperparameters** * **Epochs (num\_train\_epochs)** * Number of times the model sees the full dataset. More epochs improve learning but increase **training time** and risk **overfitting**. * **Learning rate** * Step size for optimizer updates. High learning rates can cause **unstable training**, low rates may result in **slow convergence** or underfitting. ### **Advanced Hyperparameters** * **Batch size (per\_device\_train\_batch\_size)** * Number of examples per batch before updating model parameters. Larger batch sizes **increase VRAM usage** but improve gradient estimation stability. * **Learning rate warmup steps** * Gradually increases learning rate at the start of training. Helps **stabilize training** and prevents early divergence. * **Early stopping callback** * Stops training if validation metrics do not improve after a set number of steps. Helps **prevent overfitting** and saves **compute cost**. Only necessary if training uses validation data. * **early\_stopping\_patience** determines number of evaluation calls to wait before stopping training when the metric stops improving. * **early\_stopping\_threshold** determines minimum improvement required to consider the metric as "improving”. ### Optional **Hyperparameters** * **LoRA Configuration** * Fine-tunes only a **subset of model parameters**, reducing **memory and optimizer state requirements**. * **Key LoRA parameters and their effects:** | Parameter | Description | Example / Effect | | ---------------- | -------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------ | | `r` | Rank of the low-rank matrices used for adaptation | Higher `r` can improve learning but uses more memory | | `alpha` | Scaling factor applied to the LoRA updates | `alpha = 32` → higher values amplify LoRA weight updates; often scaled relative to `r` | | `dropout` | Dropout applied to LoRA layers | `dropout = 0.05` → helps prevent overfitting by randomly zeroing some LoRA updates | | `target_modules` | List of model layers to apply LoRA | `["q_proj", "v_proj"]` → only these attention projection layers are adapted; targeting more layers increases capacity and memory use | | `fan_in_fan_out` | Whether LoRA weights are transposed for some model types | `false` for most standard models; needed for certain architectures | | `bias` | How bias terms are handled in LoRA layers | `"none"`, `"all"`, `"lora_only"`; usually `"none"` for simplicity | * Example: ```json { "use_lora": true, "lora_config": { "r": 8, "alpha": 8, "dropout": 0.05, "target_modules": ["q_proj", "v_proj"], "fan_in_fan_out": false, "bias": "none" } } ``` * **Merge Adapters** * Determines whether the LoRA adapters are **merged into the base model weights** after fine-tuning. * If true, adapters are merged into the main model, so the model can be used independently without loading separate LoRA weights. * If false, Adapters remain separate, allowing further fine-tuning or swapping different adapters. * Example: ```json { "use_lora": true, "merge_adapters": true } ``` * **Logging Strategy** * Frequency of recording training metrics, such as training loss and accuracy, to provides more insights. But it may slightly slow down training. * Options: `epoch`, `steps`, or `none`. If using steps, `logging_steps` must also be set. * Example: ```json "sft_config": { "logging_strategy": "steps", "logging_steps": 50, } ``` * **Evaluation strategy** * Determines **when validation is going to be performed** (after every epoch or a number of steps). * Options: `epoch`, `steps`, or `none`. If using `steps`, `eval_steps` must be specified (or defaults to `logging_steps`). * If different from logging strategy, evaluation logs may appear at a different interval. * Example: ```json "sft_config": { "eval_strategy": "steps", } ``` * **Mixed Precision Training** * Controls the numerical precision used during training to balance **memory usage** and **training speed**. * Options: | Options | Description | Note | | ------------------------------ | --------------------------------------------------------------- | ------------------------------------------------------- | | `fp32` (32-bit floating point) | highest precision but uses most memory | when both fp16 and bf16 are disabled | | `fp16` (16-bit floating point) | **\~50% memory reduction**, faster training | when explicitly enabled | | `bf16` (Brain Float 16-bit) | **better numerical stability** than fp16, requires Ampere+ GPUs | either explicitly enabled OR when using FlashAttention2 | * Example: ```json "sft_config": { "fp16": "false", "bf16": "true, } ``` * **Attention Implementation** * Controls which **attention algorithm** to use during training. Different implementations offer varying trade-offs between **memory usage, speed,** and **compatibility**. * Use use\_cache: false for training to save memory * Options: | Attention Type | Description | Precision | Training Speed | | ----------------------------- | ------------------------------------------------------------- | ----------- | -------------- | | eager | Universal compatibility but slowest and uses most memory | fp16/fp32 | 1x (baseline) | | sdpa | Good balance of speed and compatibility, works with fp16/fp32 | fp16/fp32 | 1.5x | | flash\_attention\_2 (default) | Fastest and most memory-efficient, but forces bf16 precision | Forced bf16 | 3x | * Example: ```json { "model_init_config": { "attn_implementation": "flash_attention_2", "use_cache": false }, } ``` * **Gradient Accumulation** * Number of batches to accumulate before a backward pass. * Useful when **batch size exceeds GPU memory**. Allows training larger effective batch sizes without running out of VRAM. * Example: ```json "sft_config": { "gradient_accumulation_steps": 4 } ``` # LLM Comparison Table This table provides a comprehensive comparison of various LLM providers and their offerings. This comparison is designed to help you make informed decisions when selecting the most suitable LLM for your needs.
ProviderModelMaximum context lengthMaximum outputPrice per 1M inputPrice per 1M output
Anthropicclaude-3-opus-20240229200,0004,096$15.00$75.00
OpenAIgpt-4-turbo-2024-04-09128,0004,096$10.00$30.00
AWS Bedrockmeta.llama3-1-405b-instruct-v1:0128,0002,048$5.32$16.00
OpenAIgpt-4o-2024-05-13128,0004,096$5.00$15.00
Googlegemini-1.5-pro2,097,1528,192$3.50$10.50
Anthropicclaude-3-5-sonnet-20240620200,0008,192$3.00$15.00
Anthropicclaude-3-sonnet-20240229200,0004,096$3.00$15.00
Coherecommand-r-plus128,0004,096$3.00$15.00
Mistralmistral-large-2407128,000$3.00$9.00
OpenAIgpt-4o-2024-08-06128,00016,384$2.50$10.00
Mistralcodestral-240532,000$1.00$3.00
AWS Bedrockmeta.llama3-1-70b-instruct-v1:0128,0002,048$0.99$0.99
AWS BedrockAmazon Titan Text Premier32,0003,072$0.50$1.50
Coherecommand-r128,0004,096$0.50$1.50
OpenAIgpt-3.5-turbo-012516,3854,096$0.50$1.50
Mistralopen-mistral-nemo-2407128,000$0.30$0.30
Anthropicclaude-3-haiku-20240307200,0004,096$0.25$1.25
AWS Bedrockmeta.llama3-1-8b-instruct-v1:0128,0002,048$0.22$0.22
AWS BedrockAmazon Titan Text Express8,192$0.20$0.60
OpenAIgpt-4o-mini-2024-07-18128,00016,384$0.15$0.60
AWS BedrockAmazon Titan Text Lite4,096$0.15$0.20
Deepseekdeepseek-chat128,0008,192$0.14$0.28
Deepseekdeepseek-coder128,0008,192$0.14$0.28
Googlegemini-1.5-flash1,048,5768,192$0.08$0.30
Coherecommand4,0964,096
Coherecommand-light4,0964,096
# Evaluation In Datasaur, we support three types of evaluation. You can choose the one that best suits your needs. * [Automated evaluation](/llm-projects/evaluation/automated-evaluation). * [Ranking](/llm-projects/evaluation/ranking-rlhf). * [Rating](/llm-projects/evaluation/rating). ## Automated evaluation Evaluate a model or pre-generated completions automatically using your preferred metrics by comparing them to a ground truth. ## Ranking Evaluate your model manually by ranking several completion results from each prompt from best to worst. You can evaluate pre-generated completions or a model and determine how many results it needs to generate. ## Rating Evaluate each completion result from each prompt by rating them with 1 to 5 stars and providing your expected completions. You can evaluate pre-generated completions or a model. # Automated Evaluation The automated evaluation feature addresses the challenges users face when manually evaluating completions. This process is time-consuming, labor-intensive, and prone to human error, leading to inconsistent evaluations. Automating the evaluation process helps users save time, improve accuracy, and ensure consistent evaluations. ## Pre-requisites To use automated evaluation, you need to complete some prerequisites based on what you want to evaluate: **To evaluate an existing model in Datasaur:** 1. Ensure the model is deployed from Sandbox. 2. Prepare a ground truth dataset in a CSV file with **two** columns: `prompt` and `expected completion`. {% file src="/files/LZSJSCQYIlMIBNy0JoDY" %} Sample file when using LLM application {% endfile %} **To evaluate pre-generated completions:** 1. Prepare a dataset in a `.csv` file with **three** columns: `prompt`, `completion`, and `expected completion`. {% file src="/files/4ajfQF4DUVVobMXwVR4A" %} Sample file when using Pregenerated dataset {% endfile %} ## Getting started 1. Navigate to the **Evaluation** page from the sidebar. 2. Click the **Create evaluation project** button and choose **Automated evaluation** project type, then **Continue**.
3. Configure your evaluation project. You can evaluate two types with automated evaluation: 1. **Model from Sandbox** * Select the model that you want to evaluate. If you can’t find your model in the list, go to the [Sandbox](/llm-projects/sandbox) where your model is created, and [deploy](/llm-projects/sandbox#deploy-models) or save to library. You can only evaluate deployed or saved models. * Upload the ground truth dataset in a CSV format containing **two** columns: `prompt` and `expected completion`.
2. **Pre-generated completions** * Upload the pre-generated completions combined with the ground truth dataset in a CSV format with three columns: `prompt`, `completion`, and `expected completion`.
4. **Manage evaluation**: Select the metric, provider, and the evaluator model you want to use for evaluation.
{% hint style="info" %} Currently, we support Langchain and Ragas as our evaluation providers. You can find the list of supported metrics in the [Evaluators](#evaluators) section. {% endhint %} ## Analyze the evaluation results After the evaluation process is completed, you can analyze the results: ### For models * **Generation cost and processing time**: View the total cost and time taken for generating completions. * **Average score**: See the overall performance score given by the evaluator. * **Detailed results**: For each prompt, you can examine: * The quality of the generated completion. * Processing time. * Individual score.
### For pre-generated completions * **Average score**: See the overall performance score given by the evaluator. * **Detailed results**: For each prompt, you can examine: * The quality of the pre-generated completion. * Individual score.
## Evaluators Automated evaluation supports various industry-standard evaluators to provide you with comprehensive insights into your model's performance. Each evaluator comes with a set of specific metrics tailored to different aspects of LLM evaluation. ### Langchain * [**Answer Correctness**](https://python.langchain.com/v0.1/docs/guides/productionization/evaluation/string/scoring_eval_chain/): Measures the accuracy of the LLM's responses compared to the ground truth. ### Ragas * [**Answer Correctness**](https://docs.ragas.io/en/latest/concepts/metrics/answer_correctness.html): Measures the accuracy of the LLM's responses compared to the ground truth. ### Deepeval * [**Answer relevance**](https://docs.confident-ai.com/docs/metrics-answer-relevancy): Evaluates how relevant the LLM's responses are to the given questions. * [**Bias**](https://docs.confident-ai.com/docs/metrics-bias): Assesses the presence of bias in the LLM's outputs based on predefined criteria. * [**Toxicity**](https://docs.confident-ai.com/docs/metrics-toxicity): Detects and quantifies toxic language or harmful content in the LLM's responses. # Multi-model evaluation This feature allows you to compare and evaluate the performance of multiple models using various metrics from evaluators like Ragas, Langchain, and Deepeval. ## Get started To evaluate multiple models: 1. Navigate to the **Evaluation** page from the sidebar. 2. Click the **Create evaluation project** button and choose **Automated evaluation** project type, then **Continue**.
3. Configure your evaluation by selecting the models to evaluate and choosing a dataset from the library. If you don’t have one, you can also upload a dataset in a CSV format containing two columns: `prompt` and `expected completion`.
{% hint style="info" %} If you can’t find your model in the list, go to the [Sandbox](/llm-projects/sandbox) where your model is created, and [deploy](/llm-projects/sandbox#deploy-models) or save to library. You can only evaluate deployed or saved models. {% endhint %} 4. Select the metric, provider, and the evaluator model you want to use for evaluation. Learn more about the [evaluators and metrics](/llm-projects/evaluation/automated-evaluation#evaluators).
5. Click **Create evaluation** **project** and wait for the evaluation process to finish. ## Analyze the evaluation results After the evaluation process is completed, you can analyze the results.
### Summary of the evaluation Here you can view the total cost, time taken for generating completions, and the overall performance score given by the evaluator.
### Result and score from each model Here you can view the quality, the score, and the processing time of the generated completions from each model.
### Evaluation details To view the evaluation details of a completion, click the three-dot menu at the far right of the row, then select **View details**.
# Custom metrics Custom metrics in automated evaluation projects let you define your own evaluation setup to fit your project’s needs. With custom metrics, you control the evaluation criteria, scoring ranges, and evaluator models to ensure accurate and meaningful assessments. ## Get started To use custom metrics: 1. Navigate to the **Evaluation** page from the sidebar. 2. Click **Create evaluation project** and select **Automated evaluation** project type, then **Continue**.
3. Configure your evaluation by selecting the models to evaluate and choosing a dataset from the library. If you don’t have one, you can also upload a dataset in a CSV format containing two columns: `prompt` and `expected completion`.
4. In step 2, change the metric to **Custom.** 5. Set up the custom metric configuration, which consists: * **Evaluator model:** The model that will evaluate the outputs of your model. * **Custom evaluator name:** Enter a unique name to identify your custom evaluation. * **Minimum and maximum score:** Define the scoring range (example: 0 to 100). If the metric uses inverted scoring, you can set the minimum value higher than the maximum (example: 100 to 0). * **Prompt:** Write a clear, detailed prompt that explains the evaluation process. Include specific criteria and instructions for assessing responses.
6. Click **Create evaluation project** and wait for your evaluation process to finish. ## Analyze the evaluation results After the evaluation process is completed, you can analyze the results. Learn more on how to [analyze the result](#analyze-the-evaluation-results). # Ranking (RLHF) ## Overview The **Ranking** evaluation project helps you assess the quality of your LLM completions using human judgment, by comparing multiple completions for the same prompt. You rank the completions from best to worst, providing insight into which outputs align most closely with your expectations. ## Prerequisites In **Ranking** projects, you can evaluate two types of completions: 1. Pre-generated completions 2. Completions generated by models from Sandbox ### **Evaluate pre-generated completions** 1. Prepare a dataset in a CSV file with several columns: `prompt` and `completion_1`, `completion_2`, `completion_3`, and so forth up to `completion_xx`. {% file src="/files/3ssA2gveoDaBX5E2xnLB" %} ### **Evaluate models from Sandbox** 1. Ensure the LLM application is deployed. 2. Prepare a dataset in a CSV file with one column: `prompt`. {% file src="/files/kV4uhIM4zyuJmW3JWnav" %} ## Create project To create Ranking evaluation projects: 1. Navigate to the **Evaluation** page under LLM Labs menu. 2. Click **Create evaluation project,** select **Ranking,** then **Continue**.
3. Set up your project. Choose what you want to evaluate with Ranking: 1. **Evaluate pre-generated completions** 1. Upload the dataset in a CSV file with several columns: `prompt` and `completion_1`, `completion_2`, `completion_3`, and so forth up to `completion_xx`.

Ranking evaluation project with pre-generated completion creation

2. **Evaluate models from Sandbox** 1. Upload the dataset in a CSV file with one column: `prompt`. 2. Select the model that you want to use to generate completions. If you can’t find your model in the list, go to the [Sandbox](https://docs.datasaur.ai/llm-projects/sandbox) where your model is created, and [deploy](https://docs.datasaur.ai/llm-projects/sandbox#deploying-the-llm) or save to library. You can only evaluate deployed or saved models.

Ranking evaluation project with LLM application creation

4. Click **Create evaluation** **project**. ## Evaluate completions Open the project to start evaluating completions. Each prompt includes at least two completions. Rank them from best to worst by dragging to reorder, then submit your answer to move on to the next prompt. {% embed url="" %} ## View evaluation results After evaluating all completions, mark the evaluation as complete from the app bar. Click the current status **Evaluation in progress** and change it to **Evaluation completed**.
After the evaluation is marked as complete, you can view the summary of the evaluation. For models from Sandbox evaluation, you can see: * Average cost and processing time for generating completions * Evaluation results in a table view
For pre-generated completions evaluation, you can see the evaluation results in a table view as well.
# Rating ## Overview The **Rating** evaluation project helps you assess the quality of your LLM outputs using human judgment, by rating and correcting the generated completions. ## Prerequisites In **Rating** projects, you can evaluate two types of completions: 1. Pre-generated completions 2. Completions generated by models from Sandbox ### **Evaluate pre-generated completions** There are two CSV formats for pre-generated completions: 1. Two column CSV format: `prompt` and `completion`. 2. Four column CSV format: `prompt_template`, `prompt`, `sources`, `completion`. {% file src="/files/5LBrx35IRg3uRUTNqQb8" %} {% file src="/files/b1MYRvWGithOfwS7UVjZ" %} ### **Evaluate models from Sandbox** 1. Ensure the model is deployed or saved to library. 2. Prepare a dataset in a CSV file with one column: `prompt`. {% file src="/files/kV4uhIM4zyuJmW3JWnav" %} ## Create project To create Rating evaluation projects: 1. Navigate to the **Evaluation** page under LLM Labs menu. 2. Click **Create evaluation project,** select **Rating,** then **Continue**.
3. Set up your project. Choose what you want to evaluate with: 1. **Evaluate pre-generated completions** 1. Upload the dataset in a CSV file with two columns: `prompt` and `completion`.
2. **Evaluate models from Sandbox** 1. Upload the dataset in a CSV file with one column: `prompt`. 2. Select the model that you want to use to generate completions. If you can’t find your model in the list, go to the [Sandbox](https://docs.datasaur.ai/llm-projects/sandbox) where your model is created, and [deploy](https://docs.datasaur.ai/llm-projects/sandbox#deploying-the-llm) or save to library. You can only evaluate deployed or saved models.
4. Click **Create evaluation project**. ## Evaluate completions Open the project to evaluate the generated completions. You should rate each completion of a prompt from 1 to 5 stars. A 5-star rating usually means the completion is already perfect, so there is no need to provide feedback or edit the completion.

Labeler mode

When the rating is below 5 stars, you have to refine the completion by providing your expected completion. After that, submit the answer to move to the next prompt.
## View evaluation results After evaluating all completions, mark the evaluation as complete from the app bar. Click the current status **Evaluation in progress** and change it to **Evaluation completed**.
After the evaluation is marked as complete, you can view the summary of the evaluation. For evaluating models from Sandbox, you can see: * Average cost and processing time for generating completions * Average evaluation score * Evaluation results in a table view
For evaluating pre-generated completions, you can see: * Average evaluation score * Evaluation results in a table view
# Performance Monitoring Performance monitoring enhances automated evaluation by letting you schedule regular evaluations of your models. Once the interval is set, evaluations run automatically, and you can check the results for timely improvements. ## Get started To use performance monitoring: 1. Navigate to the **Performance monitoring** page under **Evaluation** menu. 2. Click **Create performance monitoring**.
3. Configure your evaluation by selecting a model to evaluate and choosing a [dataset](/llm-projects/dataset) from the library. If you don’t have one, you can also upload a dataset in a CSV format containing two columns: `prompt` and `expected completion`.
4. Select the metric, provider, and the evaluator model you want to use for evaluation. Learn more about the [evaluators and metrics](/llm-projects/evaluation/automated-evaluation#evaluators).
5. In the final step, you will need to set and configure the schedule for the evaluation process. You will need to configure: 1. **Recurrence**: Choose the frequency of your evaluation. The available options are: 1. `Daily at 12:00 AM`: Your evaluation will be performed on a daily basis at midnight. 2. `Weekly on Sunday at 12:00 AM`: Your evaluation will be performed weekly on Sunday at midnight. 3. `Monthly on day 1 at 12:00 AM`: Your evaluation will be performed monthly on the first day of the month at midnight. 4. `Custom`: You can set and configure your own evaluation frequency.
2. **Monitor performance drift**: You can get notified for LLM performance drift over time. Datasaur will notify you via email when any generated completion deviates beyond a specified threshold during scheduled evaluations, indicating potential performance deterioration. 3. **Run immediately**: Evaluate your model right away after creating the project, regardless of the recurrence settings.
6. Click **Create evaluation project**, and your performance monitoring project will be created.
## Evaluation process Inside the project, you can click **Run now** to manually start the evaluation process.
Once the evaluation process has started, you will need to wait until it is completed. You'll receive an email once it's finished, or you can refresh the page to see the latest update.
## Analyze the evaluation result After the evaluation process is completed, you can analyze the results.
### Summary On the **Summary** section, you can see the cost and the processing time of the evaluation process. You can also see the average evaluator score and the performance result.
### Result and score In the results section, you can see the completions generated by your model, along with their scores for the selected metric, reasons behind the scores, and overall performance.
### Evaluation details To view the evaluation details of a completion, click the three-dot menu at the far right of the row, then select **View details**.
# Dataset ### Overview The Dataset page in LLM Labs collects all datasets available for [automated evaluation](https://docs.datasaur.ai/llm-projects/evaluation/automated-evaluation) or [fine-tuning](https://docs.datasaur.ai/llm-projects/models/fine-tuning), providing a centralized location for managing your data. ### Prerequisites Dataset must be formatted as a CSV (Comma Separated Value) file with the following columns: 1. `prompt`: This column contains the input prompt that you will feed to your LLM. 2. `expected completion`: This column holds the desired or ideal output that your LLM should generate in response to the given prompt. 3. `system_instruction` (Optional): This column contains global or contextual instructions that control how the LLM should interpret and respond to the prompt. {% file src="/files/xM4DiBmBYdfzxLKH3YqG" %} {% file src="/files/U9JQq45ugPPy8wNneCoW" %} ### Create dataset 1. Navigate to the **Dataset** menu on the left sidebar. 2. Click the **Create dataset** button.

Create dataset

3. Type the dataset name then click **Create**, and you will be redirected to the dataset table.

Dataset name

4. Click the **Upload** **dataset** button and select a .csv file containing the following columns: prompt, expected\_completion, and system\_instruction (optional).
5. Once the file is uploaded, the dataset will be automatically added to the table.

Dataset created

### Modify dataset item Once a dataset is uploaded, you can **add more**, **edit**, or **delete** dataset items. #### Add more dataset items 1. Click **Add dataset** button next to the Search field. 2. Upload the .csv file. 3. The additional datasets will be added to the table, and the existing ones remain.

Adding more dataset

#### Edit dataset item 1. Find the dataset item you want to edit using the **Search** field and right-click the dataset item to open a popover menu.
2. Then click the **Edit** option.
3. Modify the necessary details, then click **Enter** to apply the updates. Notes: For shortcut, you can simply double click the dataset item to edit the item #### Delete dataset item 1. Find the dataset item you want to edit using the **Search** field and right-click the dataset item to open a popover menu.
2. Then click the **Delete** option.
3. The dataset item will be deleted immediately. {% hint style="info" %} Please note that this action cannot be undone. {% endhint %} ### Delete entire dataset 1. In the main Dataset page, find the dataset you want to delete using the search field or filter options. Click the **More** menu (three-dots icon), then select **Delete** option.

Delete dataset option

2. Confirm the deletion by clicking the **Delete** button.
To delete multiple datasets: 1. Select the datasets and click the **Delete** button above the table.
2. Confirm the deletion by clicking the **Delete** button.
### Access via Automated evaluation Once you've created the dataset, it will be available for use in the Automated evaluation projects. [Learn more about Automated evaluation](https://docs.datasaur.ai/llm-projects/evaluation/automated-evaluation). Click **Use existing dataset** in Step 1 when creating an automated evaluation project.
A dialog will appear where you can choose the dataset to use for the project.
### Access via Fine-tuning Once you've created the dataset, it will be available for use for fine-tuning base models. [Learn more about Fine-tuning](https://docs.datasaur.ai/llm-projects/models/fine-tuning) models. Click **Use existing dataset** in Step 1 when configuring fine-tuning.
A dialog will appear where you can choose a dataset for fine-tuning.
# Pricing Plan ## Overview Datasaur recognize that every user has unique requirements. Some may need LLM capabilities occasionally for specific projects, while others may require consistent access for ongoing tasks. Our pricing models reflect this diversity, providing flexibility and cost-effectiveness. ## Pay as you go: Use what you need, pay as you go In the **Pay as you go** model, you only pay for the resources you utilize, making it a cost-effective option for infrequent or occasional usage. This model is perfect for: * **Individual users:** Experimenting with LLM capabilities for personal projects or learning purposes. * **Startups and small businesses:** Testing the value of LLMs for specific tasks before committing to a larger investment. * **Occasional users:** Requiring LLM services for infrequent or unpredictable needs. ### Details After the free trial quota is reached, **Pay as you go** becomes the default model. * **Charged usage:** You will be charged based on your actual usage of LLM Labs in these areas: * Run prompt in Sandbox. * Update embeddings in the knowledge base. * Generate completions for evaluation projects. * **Payment method:** To facilitate your **Pay As You Go** usage, kindly provide your payment credentials securely upfront through our integrated Stripe platform. This allows for convenient monthly billing based solely on your actual resource utilization within LLM Labs. * **Full feature access:** You will have access to all features within LLM Labs with the **Pay As You Go** plan. * **Limitation:** You cannot connect your own LLM credentials (for example: OpenAI keys, AWS ARN). ## Subscription: Predictable cost, consistent access The **Subscription** model offers a predictable cost structure and guaranteed access to LLM Labs features for a fixed monthly or annual fee. This model is ideal for users who: * **Require consistent LLM access:** Utilize LLMs regularly for ongoing tasks or projects. * **Benefit from predictable costs:** Prefer a fixed monthly or annual fee for budgeting purposes. * **Value guaranteed access:** Want to ensure uninterrupted access to LLM Labs features. ### Details * **Enrollment:** You need to contact our sales team at to use the **Subscription** plan. * **Access:** All features of LLM Labs are available, including the ability to connect and manage personal LLM credentials. * **LLM Credentials:** You can provide and utilize their own LLM credentials. * **Pay as you go integration:** You can leverage the **Pay as you go** model on top of their subscription for additional usage needs. ## Choose the right plan The best plan for you depends on your specific needs and usage patterns. Consider the following factors: 1. **Frequency of use:** How often will you need to use LLM Labs? 2. **Predictability of usage:** Can you anticipate your LLM usage needs in advance? 3. **Budget constraints:** Do you have a fixed budget for LLM services? Datasaur is committed to providing a user-friendly and cost-effective platform for all. We encourage you to explore our **Pay as you go** and **Subscription** plans to determine the best fit for your requirements. {% hint style="info" %} If you have any questions, please don't hesitate to contact our support team at . {% endhint %} # Workspace A workspace is the main environment where your team collaborates and manages labeling work. It includes everything needed, such as projects, members, roles, analytics, and other supporting settings. ## Create a team After signing in, you are directed to your personal workspace. To create a team workspace: 1. Click **Switch workspace** in the top-right corner. 2. Select **Create a new workspace**. 3. Enter a team name and upload a logo (max 500 KB).
You can update the team name and logo later from **Workspace settings**.
## Members ### Invite members To invite members: 1. Go to the **Members** page. 2. Click **Invite member**. 3. Enter up to 20 email addresses (separated by commas). 4. Select a [role](/workspace-management/role-and-permission). 5. Click **Send invitation**. Invited members will receive an email with instructions to join. ![](/files/8vye9EHvStUSE9hxSkpT) ### Update members' role To update a member’s role, click the **three-dot menu** next to their name and select a new role. ![](/files/lXNB6aO7rH5cyEyoHPbE) You can also sort and filter members by name or role. ![](/files/1mKSuyeihz2WKdeA5OfH) ![](/files/KDXC7nTosUFQjEbHMqi1) ### Remove members To remove a member, click the triple-dot menu on a member and select **Remove from workspace**.
{% hint style="info" %} Removing a team member from the workspace does not affect their labeling work. {% endhint %} ## Projects Admins and supervisors can create projects in a team workspace. Learn the details on the [Create a project](/data-studio-projects/creating-a-project) page. Admins, supervisors, and reviewers have access to all projects in reviewer mode. Learn the details on [Review projects](/data-studio-projects/reviewing-nlp-projects) page. ## Manage tags You can define tags in your workspace and reuse them when creating and managing projects. Only admins can manage tags. ![](/files/MXqelirUdIs6vgWzKbL9) ## Manage export settings You can control which roles can export data by method. There are 3 methods: * **Download:** Allow labelers/reviewers to exports the file directly to their device. * **Send to my email:** Allows labelers/reviewers to receive exported files via email. * **Send to my external storage:** Allows labelers/reviewers to export files to a connected external storage service. ![](/files/7eUk0iNqzqGjpvONH5iM) For example, if labelers are not allowed to export at all, the **Export** option will not appear in the **File** menu. ![](/files/gUb6a2ZfovhzHOqJcf2i) # Role & Permission ### Overview Roles and permissions help you manage access and responsibilities across the workspace and within projects. There are two types of roles: * Workspace-level roles * Project-level roles ### Workspace roles Workspace roles define a member’s overall access and permissions across all projects in the workspace. There are 4 roles:
RoleDescription
AdminHas full access to the workspace and all projects.
SupervisorResponsible for managing labelers and reviewers, and handling daily operations.
ReviewerReview labeling tasks. Reviewers can view all projects, including both assigned and unassigned ones.
LabelerPerform assigned tasks. Labelers can view view only assigned projects.
To assign a workspace role: 1. Go to the **Members** page. 2. Click the **three-dot menu** next to a member. 3. Update the role. Refer to the table below to understand the differences in permissions between the 4 roles.
PermissionLabelerReviewerSupervisorAdmin
Workspace Management
Update workspace informationv
Modify export permissionv
Enable/disable notificationsv
View project activitiesvv
Member Management
View all membersvvvv
View member detailsvv
Invite members to a Workspacevv
Change member rolev
(can’t manage admins or promote member as a supervisor)		v
Delete members from Workspacev
(can’t delete admin or supervisor)		v
Project Management
View all projectsAssigned onlyvvv
Create projectvv
Rename/delete projectOwned onlyv
Modify project settingsv
Archive/restore projectOwned onlyv
View all project templatesvv
Create/save/edit project templatev
Share projectAssigned onlyvvv
Export projectAssigned onlyvvv
Create/update/delete project tagv
Label and Question Management
View all label setsvv
Create label setvv
Update/delete label setOwned onlyv
View all question setsvv
Create question setvv
Update/delete question setOwned onlyv
Data Labeling Workflow
Assign labeling and reviewing task to membersvv
Access the Labeler mode to perform labeling tasksAssigned onlyAssigned onlyAssigned onlyAssigned only
Review labeling resultAssigned onlyvvv
Create/update/delete commentsAssigned onlyvvv
Assignable Project Role
Labeler onlyv
Reviewer onlyvvvv
Labeler & Reviewervvvv
Advanced
Create/update/delete actionv
Create/update/delete file transformerv
Create/delete external object storagev
Create/delete API keysv
Enable/disable SAML and SCIMv
Analytics
View team overviewvv
View Inter-Annotator Agreementvv
Export Inter-Annotator Agreementvv
Configure custom report buildervv
Export/download custom report buildervv
Export/download chartvv
View Evaluation Metricsvv
### Project roles Project roles define responsibilities within a specific project. Permissions follow the workspace role first, and the workspace role cannot be modified at the project level. There are 2 roles:
RoleDescription
ReviewerReview labeling tasks through Reviewer Mode.
LabelerPerform labeling tasks through Labeler Mode.
You can assign project roles during project creation or update them later in **Project settings**. Refer to the table below to understand the differences in permissions between the roles.
PermissionLabelerReviewerReviewer (with Supervisor Workspace Role)Reviewer (with Admin Workspace Role)
Modify personalization settingsvvvv
Modify project settingsv
Modify project assignment settingsvv
Modification
Modify textIf permitted in project settingsIf permitted in project settingsIf permitted in project settingsv
Modify label setIf permitted in project settingsvvv
Modify question setIf permitted in project settingsvvv
Data Labeling Workflow
Assign labeling and reviewing task to membersvv
Access the Labeler mode to perform labeling tasksv
Apply labelsAssigned onlyvvv
Review labeling resultvvv
Create/update/delete commentsvvvv
Upload/use existing guidelinesAssigned onlyvvv
Enable/disable extensionsAssigned onlyvvv
# Analytics In a team workspace, there are several ways to view project progress and members’ labeling performance. The **Analytics** pages are exclusively accessible to administrators. For convenient data access, you can choose to export it, and it will be delivered to your email. ## Charts Use the following guidelines to interpret the charts: * Higher values generally indicate better performance (speed, accuracy, and efficiency). * The **Throughput** and **Quality** charts may differ: * **Throughput** includes labels applied in reviewer mode * **Quality** includes only labels from labeler mode * Large differences may indicate heavy reviewer involvement * High **Efficiency** with low **Throughput** and **Quality** often occurs in projects with significant pre-labeled data, since those metrics exclude pre-labeled labels. ### Overall projects Displays the current total projects distribution based on its status. This is a snapshot (not time-series data). ![](/files/BPAzOzT2RsIyb1pS2mtE) ### Remaining files Displays the current remaining files from incomplete projects, grouped by project status. This is a snapshot (not time-series data). ![](/files/3S1PjdaASVkmutAgYFV9) ### Throughput Measures how quickly annotations are produced. It is calculated from: * Labels applied by labelers * Labels manually applied in reviewer mode and manual conflict resolution (excluding labels automatically accepted through consensus) ![](/files/WTOHx8vEzybpkxY0PPQA) ### Efficiency Measures how effectively accepted labels are produced over time. It is calculated by dividing accepted labels from reviewer mode by the total time spent by all team members, measured daily. The accepted labels include manually reviewed labels, consensus labels, labels applied by reviewers, and pre-labeled data. ![](/files/8YFXGrVSqRYRmm2hKLdc) ### Quality It breaks down conflicts between labelers and shows how reviewers resolved them. This metric also provides insight into labeler agreement (especially when combined with [IAA](/workspace-management/analytics/inter-annotator-agreement)) and reviewer effectiveness. {% hint style="info" %} Not applicable to bounding box labeling projects. {% endhint %} The data is categorized as follows: 1. **Accepted labels:** Labels accepted through consensus or manual review. 2. **Rejected labels:** Labels rejected through consensus or manual review. 3. **Unresolved conflicts:** Labels not yet accepted or rejected. ![](/files/uSxydNieTQpYPF4ttqjr) ### Cumulative time spent Tracks total time spent per day by labelers and reviewers. How it works: * This metric is calculated only when the project tab is active. Time is not counted when you switch tabs. * Idle time (no mouse, keyboard, or scroll activity for 2 minutes) is excluded. * When a project is open and active, it will count the time spent by sending a request to the backend every 60 seconds. * If a workspace becomes idle before 60 seconds have passed, it will send the elapsed time since the last request to the backend. Time spent using AI-assisted features is included, as it is considered part of the workflow. ![](/files/30Zv42sIRT5AJplgl5zs) ## Ways to view the data ### Analytics overview The **Analytics > Overview** page provides a high-level summary across all projects, including [inter-annotator agreement](/workspace-management/analytics/inter-annotator-agreement). You can use the project tag filter to view detailed analytics for specific projects. Multiple tags use `OR` logic (projects matching any selected tag are included).
### Project analytics In the **Projects** page, you can view the project details from the **Name** column, such as: * Total files * Total time spent * Token count (span labeling) * Row count (row labeling). You can also view labelers' progress by hovering on their avatar in the **Members** column, and view the summary of your labeling progress by hovering your labeling status in **Status** column. For detailed analysis of a specific project, click the triple-dot menu and select the **View project analytics**. Learn the details on the [Project report](/workspace-management/analytics/project-report) page.
### Member analytics
On the **Members** page, you can manage members and view each member’s performance. To access a member’s analytics page, open the three-dot menu and select **View member details**. You can view their performance as a reviewer or labeler by switching between the corresponding tabs. In each tab, there will be **Throughput** and **Efficiency** charts, along with a table of their assigned projects and labeling analytics. ### Analytics extension in a project The **Analytics** extension is useful to track each labeler’s progress during an ongoing project, including: * Number of labels produced * Number of questions answered * Remaining documents per labeler Pre-labeled data is also included in each labeler’s metrics. For more details about the extension, refer to the [Analytics extension](/advanced/extensions/analytics) page. # Inter-Annotator Agreement (IAA) Inter-annotator agreement (IAA) measures how consistently multiple annotators make the same labeling decisions for a given label category or class. It helps evaluate the clarity of your guidelines and the reproducibility of your results. We support two algorithms to calculate the agreement between two annotators: * [Cohen's Kappa](/workspace-management/analytics/inter-annotator-agreement/cohens-kappa-calculation) * [Krippendorff's Alpha](/workspace-management/analytics/inter-annotator-agreement/krippendorffs-alpha-calculation) Note that we apply the scale interpretation of Krippendorff's Alpha for both methods as in Image 1. ![Krippendorff's alpha scale](https://docs.google.com/drawings/u/0/d/skeGno0Afrw0ZIKAZpYbZ2A/image?w=513\&h=93\&rev=137\&ac=1\&parent=1UmG4KLgmliNMX-9QkW8wSX64xkFfe-GlU7jp5IM8lDI) * **Discard** will be presented in red. * **Tentative** will be presented in yellow. * **Good** will be presented in green. IAA is calculated in the background as soon as a project status changed to **Ready for review** (after all labelers mark the project as complete) or **Complete** (a reviewer marks the project as complete). ## View IAA data You can view inter-annotator agreement at both the workspace and project levels to analyze consistency across your labeling work. To view IAA across projects: 1. Go to **Analytics > Team Overview** from the left sidebar. 2. Open the **Inter-Annotator Agreement** tab. Only **admin** can view this data. ### Workspace level At the workspace level, IAA is calculated as a weighted average across all projects. It does not recalculate all labels or answers. Instead, it considers how often each label or answer occurs between annotators, providing a more accurate representation of overall agreement.

Team overview IAA

### Project level To view IAA at the project level, select a project from the **Project** field.
You can also view it from the **Project analytics** page: 1. Go to the **Projects** page. 2. Click the **three-dot menu** on a project and select **View project analytics**. 3. Open the **Inter-Annotator Agreement** tab. # Cohen's Kappa Calculation Explain how Datasaur implements the Cohen's Kappa algorithm. [Cohen's Kappa](https://en.wikipedia.org/wiki/Cohen's_kappa) is one of the algorithms that is supported by Datasaur to calculate the agreement while taking into account the possibility of chance agreement. This section explains how labels from labelers and reviewers are processed into an agreement matrix and used to compute Cohen’s Kappa.
## Sample data Suppose there are 2 labelers: Labeler A and Labeler B, who labeled the same sentences. ![Labeler A](/files/vzeXRZAUFjuwCMMLQN6C) ![Labeler B](/files/sxVnm8hCNDJBh3Fl2AcE) There is also a reviewer who labeled the same sentences. ![Reviewer](/files/YeKsPDlk3nry560V27aB) ## Calculating the data ### Agreement records Based on the screenshots above, we map those labels into the agreement records below: ### **Agreement table/confusion matrix** The agreement records are then converted into a confusion matrix. For this example, the matrix is constructed using data from Labeler A and Labeler B. ![](/files/f8E2rMfySEkX9vqvmh0R) ### Calculating the Kappa From the matrix above, there are **7** records with **4** agreements. ![](/files/-MbjYJooXhsKEIXkvCE0) The observed proportionate agreement is: ![](/files/-MbjYJop2IZQ3GROglNg) To calculate the probability of random agreement, we note that: * Labeler A labeled `EVE` once and Labeler B didn't label `EVE`. Therefore, the probability of random agreement on the label `EVE` is: ![](/files/voqPzphxcDRdsuFfbUwP) * Compute the probability of random agreement for all labels: ![](/files/-MbjYJormS9xj_2DE282) The full random agreement probability is the sum of the probability of random agreement for all labels: ![](/files/-MbjYJos19gO4G0E6zKB) Finally, we can calculate the Cohen's Kappa: ![](/files/-MbjYJotnA6mMLbDJk-k) The Kappa value for Labeler A and Labeler B is 0.49. #### **Kappa for Labeler A and Reviewer** With the same calculation, the Kappa value for Labeler A and the reviewer is 0.36. ![](/files/uYZhwykZn25xuFy0S14x) #### **Kappa for Labeler B and Reviewer** With the same calculation, the Kappa value for Labeler B and the reviewer is 0.475. ![](/files/kGfToI5BXZHvO5E1I7tI) ![](/files/lvx1e98p4uXISa2qk1vP) ## Summary * Missing labels from a labeler are treated as empty labels. * Chance agreement depends on: * The number of labels in a project. * The number of label classes. * When both labelers agree but the reviewer rejects the labels: * The agreement between the two labelers increases. * The agreement between the labelers and the reviewer decreases. # Krippendorff's Alpha Calculation Explain how Datasaur implements the Krippendorff's Alpha algorithm. ## Sample data Suppose there are 2 labelers and 1 reviewer: Labeler A, Labeler B, and Reviewer, who labeled the same spans. Labeler A's work is visualized in image 1, Labeler B's work is visualized in image 2, and Reviewer's work is visualized in image 3. ![Labeler A's work](/files/vzeXRZAUFjuwCMMLQN6C) ![Labeler B's work](/files/sxVnm8hCNDJBh3Fl2AcE) ![Reviewer's work](/files/YeKsPDlk3nry560V27aB) ## Calculate the agreement In this section, we will see the calculation details between Labeler A and Reviewer. ### 1. Arrange the data First, we need to arrange the sample data into a table for better visualization. | Span | Labeler A | Reviewer | | --------------------- | --------- | -------- | | The Tragedy of Hamlet | EVE | TITLE | | Prince of Denmark | PER | | | Hamlet | PER | PER | | William Shakespeare | PER | PER | | 1599 | YEAR | YEAR | | 1601 | YEAR | YEAR | | Shakespeare | ORG | PER | | 30557 | | QTY | ### 2. Clean the data Second, spans with only have 1 label (`Prince of Denmark` and `30557`) should be removed, as they can introduce errors in the calculation. After cleaning, the results still reflect the agreement level between the two annotators. The cleaned data is shown in the table below.
SpanLabeler AReviewerReviewer
The Tragedy of HamletEVETITLETITLE
HamletPERPERPER
William ShakespearePERPERPER
1599YEARYEARYEAR
1601YEARYEARYEAR
ShakespeareORGPERPER
### 3. Create the agreement table Third, we need to create an agreement table based on the cleaned data. The table is visualized in Table 3.

Table 3. Agreement table

Based on the table, 5 values are calculated: $$n$$, $$r\_i$$, $$r\_k$$, $$r$$, and $$r'$$. #### Total spans in the data * $$n$$ is the total spans in the data. * Here, $$n=6$$ because there are 6 spans. #### Total labels in each span $$ r\_i=\sum\limits\_{k=1}^{m}r\_{ik} (1) $$ * $$r\_i$$ is the total labels that span $$i$$ has. * $$m$$ is the total number of label. * Here, $$m=5$$ because there are 5 labels. * $$r\_{ik}$$ is the number of $$k$$ label in span $$i$$. Here is the calculation result. * $$r\_1=r\_{1,EVE}+r\_{1,ORG}+r\_{1,PER}+r\_{1,TITLE}+r\_{1,YEAR}=1+0+0+1+0=2$$ * $$r\_2=r\_{2,EVE}+r\_{2,ORG}+r\_{2,PER}+r\_{2,TITLE}+r\_{2,YEAR}=0+0+2+0+0=2$$ * $$r\_3=r\_{3,EVE}+r\_{3,ORG}+r\_{3,PER}+r\_{3,TITLE}+r\_{3,YEAR}=0+0+2+0+0=2$$ * $$r\_4=r\_{4,EVE}+r\_{4,ORG}+r\_{4,PER}+r\_{4,TITLE}+r\_{4,YEAR}=0+0+0+0+2=2$$ * $$r\_5=r\_{5,EVE}+r\_{5,ORG}+r\_{5,PER}+r\_{5,TITLE}+r\_{5,YEAR}=0+0+0+0+2=2$$ * $$r\_6=r\_{6,EVE}+r\_{6,ORG}+r\_{6,PER}+r\_{6,TITLE}+r\_{6,YEAR}=0+1+1+0+0=2$$ #### Total of each label $$ r\_k=\sum\limits\_{i=1}^{n}r\_{ik} (2) $$ * $$r\_k$$ is the total of $$k$$ label in the data. * $$n$$ is the total spans in the data. * $$r\_{ik}$$ is the number of $$k$$ label in span $$i$$. Here is the calculation result. * $$r\_{EVE}=r\_{1,EVE}+r\_{2,EVE}+r\_{3,EVE}+r\_{4,EVE}+r\_{5,EVE}+r\_{6,EVE}=1+0+0+0+0+0=1$$ * $$r\_{ORG}=r\_{1,ORG}+r\_{2,ORG}+r\_{3,ORG}+r\_{4,ORG}+r\_{5,ORG}+r\_{6,ORG}=0+0+0+0+0+1=1$$ * $$r\_{PER}=r\_{1,PER}+r\_{2,PER}+r\_{3,PER}+r\_{4,PER}+r\_{5,PER}+r\_{6,PER}=0+2+2+0+0+1=5$$ * $$r\_{TITLE}=r\_{1,TITLE}+r\_{2,TITLE}+r\_{3,TITLE}+r\_{4,TITLE}+r\_{5,TITLE}+r\_{6,TITLE}=1+0+0+0+0+0=1$$ * $$r\_{YEAR}=r\_{1,YEAR}+r\_{2,YEAR}+r\_{3,YEAR}+r\_{4,YEAR}+r\_{5,YEAR}+r\_{6,YEAR}=0+0+0+2+2+0=4$$ #### Total labels in the data $$ r=\sum\limits\_{i=1}^nr\_i (3) $$ * $$r$$ is the total labels in the data. * $$n$$ is the total spans in the data. * $$r\_i$$ is the total labels that span $$i$$ has. Here is the calculation result. * $$r=r\_1+r\_2+r\_3+r\_4+r\_5+r\_6=12$$ #### Average number of labels per span $$ r'=\frac{r}{n} (4) $$ * $$r'$$ is the average number of labels per span. * $$n$$ is the total spans in the data. Here is the calculation result. * $$r'=\frac{r}{n}=\frac{12}{6}=2$$ ### 4. Choose weight function Fourth, we need a weight function to weight the labels. Every label is treated equally because one label is no different from the other. Hence, the weight function that will be used is stated in Formula (5). $$ w\_{ik}=r\_{ik} (5) $$ * $$w\_{ik}$$ is the weighted number of $$k$$ label in span $$i$$. * $$r\_{ik}$$ is the number of $$k$$ label in span $$i$$. ### 5. Calculate Pa Fifth, the observed weighted percent agreement is calculated. #### Weighted number of labels We will start by calculating the weighted number of labels using Formula (6). $$ r\_{ik+}=\sum\limits\_{l=1}^{m} w\_{kl}r\_{il} (6) $$ * $$r\_{ik+}$$ is the weighted number of $$k$$ label in span $$i$$. * $$m$$ is the total number of label. * $$w\_{kl}$$ is the weighted number of $$l$$ label in span $$k$$. * $$r\_{il}$$ is the number of $$l$$ label in span $$i$$. For example, we can apply Formula (6) to calculate the weighted EVE label in span 1. $$ r\_{1,EVE+}=\sum\limits\_{l=1}^{5} w\_{EVE,l}r\_{1,l}=1*1+0*0+0*0+0*1+0\*0=1 $$ We need to calculate all the span and label combinations. The complete calculation result is visualized in Table 4.

Table 4. Weighted number of labels

#### Agreement percentage After we got the weighted number of labels, we need to calculate the agreement percentage for a single span and label using Formula (7). $$ p\_{a|ik}=\frac{r\_{ik}(r\_{ik+}-1)}{r'(r\_i-1)} (7) $$ * $$p\_{a|ik}$$ is the agreement percentage of $$k$$ label in span $$i$$. * $$r\_{ik}$$ is the number of $$k$$ label in span $$i$$. * $$r\_{ik+}$$ is the weighted number of $$k$$ label in span $$i$$. * $$r'$$ is the average number of labels per span. * $$r\_i$$ is the total labels that span $$i$$ has. For example, we can apply Formula (7) to calculate the agreement percentage of EVE label in span 1. $$ p\_{a|1,EVE}=\frac{r\_{1,EVE}(r\_{1,EVE+}-1)}{r'(r\_1-1)}=\frac{1(1-1)}{2(2-1)}=0 $$ We need to calculate all the span and label combinations. The complete calculation result is visualized in Table 5.

Table 5. Agreement percentage

#### Agreement percentage of a single span We can simplify the result by getting the agreement percentage of a single span using Formula (8). $$ p\_{a|i}=\sum\limits\_{k=1}^{m} p\_{a|ik} (8) $$ * $$p\_{a|i}$$ is the agreement percentage of span $$i$$. * $$m$$ is the total number of label. * $$p\_{a|ik}$$ is the agreement percentage of $$k$$ label in span $$i$$. For example, we can apply Formula (8) to calculate the agreement percentage of span 1. $$ p\_{a|1}=\sum\limits\_{k=1}^{5} p\_{a|1,k}=0+0+0+0+0=0 $$ We need to calculate the agreement percentage of all spans. The complete calculation result is visualized in Table 6.

Table 6. Agreement percentage of eacpan

#### Average agreement percentage From the previous calculation, we can calculate the average agreement percentage using Formula (9). $$ p\_a'=\frac{1}{n}\sum\limits\_{i=1}^{n}P\_{a|i} (9) $$ * $$p\_a'$$ is the average agreement percentage. * $$n$$ is the total spans in the data. * $$p\_{a|i}$$ is the agreement percentage of span $$i$$. We can apply Formula (9) to calculate the average agreement percentage. $$ p\_a'=\frac{1}{6}\sum\limits\_{i=1}^{6}P\_{a|i}=\frac{1}{6}(0+1+1+1+1+0)=0.6666 $$ #### Calculate Pa Finally, the observed weighted percent agreement is calculated using Formula (10). $$ p\_a=p\_a'(1-\frac{1}{nr'})+\frac{1}{nr'} (10) $$ * $$p\_a$$ is the observed weighted percent agreement. * $$p\_a'$$ is the average agreement percentage. * $$n$$ is the total spans in the data. * $$r'$$ is the average number of labels per span. We can apply Formula (10) to calculate the observed weighted agreement percentage. $$ p\_a=p\_a'(1-\frac{1}{nr'})+\frac{1}{nr'}=0.6666(1-\frac{1}{6\times2})+\frac{1}{6\times2}=0.6944 $$ ### 6. Calculate Pe Sixth, the chance weighted percent agreement is calculated. #### Classification probability We start by calculating the classification probability for each label using Formula (11). $$ \pi\_k=\frac{r\_k}{r} (11) $$ * $$\pi\_k$$ is the classification probability for $$k$$ label. * $$r\_k$$ is the total of $$k$$ label in the data. * $$r$$ is the total labels in the data. Here is the calculation result. * $$\pi\_{EVE}=\frac{r\_{EVE}}{r}=\frac{1}{12}=0.0833$$ * $$\pi\_{ORG}=\frac{r\_{ORG}}{r}=\frac{1}{12}=0.0833$$ * $$\pi\_{PER}=\frac{r\_{PER}}{r}=\frac{5}{12}=0.4166$$ * $$\pi\_{TITLE}=\frac{r\_{TITLE}}{r}=\frac{1}{12}=0.0833$$ * $$\pi\_{YEAR}=\frac{r\_{YEAR}}{r}=\frac{4}{12}=0.3333$$ #### Calculate Pe To calculate the chance weighted percent agreement, Formula (11) can be applied to Formula (12). $$ p\_e=\sum\limits\_{k=1}^{m}{\pi\_k}^2 (12) $$ * $$p\_e$$ is the chance weighted percent agreement. * $$m$$ is the total number of label. * $$\pi\_k$$ is the classification probability for $$k$$ label. Here is the chance weighted percent agreement calculation. $$p\_e=\sum\limits\_{k=1}^{m}{\pi\_k}^2$$ $$p\_e={\pi\_{EVE}}^2+{\pi\_{ORG}}^2+{\pi\_{PER}}^2+{\pi\_{TITLE}}^2+{\pi\_{YEAR}}^2$$ $$p\_e=0.0833^2+0.0833^2+0.4166^2+0.0833^2+0.3333^2$$ $$p\_e=0.3055$$ ### 7. Calculate the Alpha Finally, Krippendorff's alpha is calculated using Formula (13). $$ \alpha=\frac{p\_a-p\_e}{1-p\_e} (13) $$ * $$\alpha$$ is the Krippendorff's alpha between Labeler A and Reviewer. * $$p\_a$$ is the observed weighted percent agreement. * $$p\_e$$ is the chance weighted percent agreement. We can get the $$\alpha$$ by applying $$p\_a$$ and $$p\_e$$ to Formula (13). $$ \alpha=\frac{p\_a-p\_e}{1-p\_e}=\frac{0.6944-0.3055}{1-0.3055}=0.56 $$ ## Summary * We apply the same calculation for agreement between labelers and between the reviewer and labelers. * Missing labels from a single labeler will be removed. * Chance agreement depends on: * The number of labels in a project. * The number of label options. * When both labelers agree but the reviewer rejects the labels: * The agreement between the two labelers increases. * The agreement between the labelers and the reviewer decreases. # Custom Report Builder Build your own report using a group table, metrics, and segments. You can generate a customized CSV report by selecting which data columns to include and filtering out unnecessary data. {% hint style="info" %} Exporting a report may take time and depends heavily on the amount of data generated. {% endhint %} ## Terms and definitions ### Group table A group table is a way to categorize combinations of metrics and segments. There are 4 group tables: * **Labeling table**: Reports labeling activity and performance, including label counts, conflicts, and quality measures. * **Time spent table**: Reports time spent across documents, labels, or rows. * **Progress table**: Reports completion progress. * **Inference cost table**: Reports the cost of AI inference incurred during automated labeling. Not every metric can be combined with every segment. Each metric has a scope, such as document-level or label-level. In most cases, it only makes sense to combine attributes that share the same scope. For example, **Time spent** metric (in **Time spent table** group) is a document-level metric, so it can only be used up to document-level granularity segments, such as **Document name***.* It is not meaningful to combine it with label-level segments. ### Metrics Metrics are quantitative measurements associated with specific segments. You can hover over each metric to see a detailed explanation. In the report, metric columns are placed after segment columns. ### Segments Segments are used to view and classify quantitative data. They act as the base columns for calculating metrics. You can hover over each segment to see a more detailed explanation. In the report, segment columns appear before metric columns. #### Team member In most cases, the email field represents a specific user. However, this value may sometimes be empty due to the following reasons: * Consensus-based labeling in reviewer mode: The labels or answers were automatically applied based on consensus. * Unresolved conflicts in reviewer mode: The labels are still in conflict and have not yet been accepted or rejected, so no user is attributed. ### Filter Filters are optional but highly recommended to remove unnecessary data and improve report generation speed. You can filter data using segments only. Regardless of which segments you include in the report, you can still use any segment as a filter. ## Generate a report 1. Click the **Configure** button.
2. Select a group table.
3. Select columns for both metrics and segments.
4. Filter by dates, projects, tags, labeling types, and members as needed.
5. Click **Save**. 6. A preview of the data will be generated for you. Use this preview to see and make sure you have the correct table structure of your report.
7. Click **Export** to send the report to your email inbox. Alternatively, click **Download** to obtain the report directly to your device.
# Project Report See the summary of the labeling work for a specific project ## Summary dialog
When a project is completed in reviewer mode, a summary dialog displays key metrics summarizing the labeling results. This feature is accessible only to **admins** and **supervisors**. You can also export the project or view the full report from this dialog. If you frequently reopen a completed project, you can choose not to show the summary dialog again for that project. ## Report tab in Project Analytics page
This tab provides a more comprehensive report on labeling results. All calculations are based on the reviewer mode of the project. Some metrics may be partially available if the project is not yet completed. The report starts with the same metrics shown in the summary dialog. Each metric has a tooltip that explains how it is calculated. It also includes an IAA (inter-annotator agreement) summary, highlighting agreement levels between reviewers and labelers. Finally, the report displays the most frequently used label classes or most selected answers, depending on the project's labeling type: * **Top label classes:** Available for both span and bounding box labeling. * **Top selected answers:** Available for both row and document labeling. Only questions with predefined options are included (radio button, dropdown, and hierarchical dropdown). # Evaluation Metrics Assess labelers’ answers using reviewers’ labels as the ground truth. To view evaluation metrics for a project: 1. Go to the **Projects** page. 2. Click the three-dot menu on a specific project and select **View project analytics**. 3. Go to the **Evaluation metrics** tab. {% hint style="info" %} Evaluation metrics are available only after a **project is completed**. During calculation, labels from **reviewer mode** are treated as the ground truth, and each labeler is evaluated against this version. {% endhint %} Currently, evaluation metrics are available for: * **Row labeling** projects with dropdown, hierarchical dropdown, True/False, or single choice questions that have a single answer (excluding multiple answers for dropdown questions). * **Span labeling** projects. ## How to use Evaluation metrics are typically used to assess models rather than human labeling quality. This differs from inter-annotator agreement (IAA), which measures agreement between labelers. In this system, reviewer mode acts as the ground truth and is compared against each labeler. You can evaluate a model using one of the following approaches: * Use a labeling agent. * Use the **ML-assisted labeling** extension. * Use pre-labeled data (Datasaur schema). ### Labeling agent The [labeling agent](/agent/labeling-agent) automatically calculates evaluation metrics after the project is completed by a reviewer. ### ML-assisted labeling Use [ML-assisted labeling](/advanced/extensions/ml-assisted-labeling) to integrate model inference results. A designated labeler **runs the model**, and the responses represent the model’s predictions. These responses are then compared to the reviewer’s answers for evaluation. This approach supports evaluating multiple models. To evaluate models with the **ML-assisted labeling** extension: 1. Create a project with no consensus to allow reviewers to set the correct answers directly. 2. Assign one person as both labeler and reviewer (optional, for simplicity). To evaluate multiple models, assign additional labelers accordingly. 3. Open the project and ensure you are in the labeler mode. 4. Open the **ML-assisted labeling** extension and call the model. Accept all the suggestions. 5. Switch to reviewer mode and finalize the ground truth. 6. Mark the project as complete. This will trigger the evaluation metrics calculation. ### Pre-labeled data This approach evaluates a single model using pre-labeled data with the[ Datasaur Schema format](/compatibility-and-updates/supported-formats). The pre-labeled data is then compared to the reviewer’s answers for evaluation. To evaluate a model using pre-labeled data: 1. Create a project with pre-labeled data in Datasaur schema format. 2. Assign one person as both labeler and reviewer (optional, for simplicity). 3. Open the project. 4. Switch to reviewer mode and finalize the ground truth. 5. Mark the project as complete. This will trigger the evaluation metrics calculation. ## Metrics Evaluation metrics are calculated using standard algorithms. The process begins with constructing a confusion matrix, and results are macro-averaged to produce final scores. ### Handling empty labels 1. In row labeling projects, empty answers (unanswered optional questions) are included in the calculation, as they may represent valid outcomes. 2. In span labeling projects, empty spans are excluded due to the large number of possible unlabeled spans, which would skew results. ### Accuracy * The proportion of correctly labeled instances among the total labels. * Calculation = total correct labels divided by total labels. ### Precision * The ratio of correctly labeled positive instances to the total instances predicted as positive. * Calculation = True Positives / (True Positives + False Positives). * *"Of all the instances predicted as positive, how many were actually positive?"* * Real world example: Spam detection, where false positives should be minimized, because we do not want to have a perfectly normal email being incorrectly classified as a spam. ### Recall * The ratio of correctly labeled positive instances to the total actual positive instances. * Calculation = True Positives / (True Positives + False Negatives). * *"Of all the actual positive instances, how many were correctly identified?"* * Real world example: Medical diagnosis, where false negatives must be minimized, because the system cannot afford to label a cancerous case as a non-cancerous one. ### F1 score * The harmonic mean of precision and recall. * Calculation = 2 \* ((Precision \* Recall) / (Precision + Recall)). * A score of 1 indicates a perfect balance between precision and recall. ### Confusion matrix A confusion matrix provides a detailed breakdown of predictions versus ground truth. * Row labeling: * The matrix represents each eligible question. * Both the rows and columns correspond to the available options for each question. * An additional *empty answer* row and column are included to account for unanswered optional questions, as described above. * Span labeling: * The matrix represents a specific label set. * Both the rows and columns correspond to the label classes applied within that set. * An additional *empty label* row and column are included. The empty label pair is displayed as **N/A**, consistent with the explanation provided in the Metrics section. ## Filters By default, metrics are calculated across all data. But, the metrics can be filtered by: 1. By documents. 2. By labelers. 3. By questions (row labeling only). 4. By label sets (span labeling only). # Activity The **Activity** page tracks all kinds of actions within a workspace.
## Tracked activities ### Project lifecycle 1. **Project creation:** When a member creates a project. 2. **Project rename:** When a member renames a project. 3. **Initial project access:** The first time a member opens a project in labeler mode or reviewer mode. 4. **Project status changes:** Each time a reviewer changes the status in reviewer mode. 5. **Labeling status changes:** Each time a labeler changes the status of a project in labeler mode. 6. **Project export:** When an entire project (all documents) is exported. 7. **Project archive or restore:** When a member archives or restores a project. 8. **Project deletion:** When a member deletes a project. 9. **Document export:** When a specific document is exported from a project. 10. **Document addition:** When documents are added to an existing project. 11. **Labeling guideline update:** When a member adds or modifies the labeling guideline. 12. **Project configuration changes:** When a member modifies the assignment, task settings, or admin settings of a project. ### Labeling activities #### Span labeling, conversational labeling, bounding box labeling 1. **Single label application:** When a member applies a label manually, it counted separately for each member in labeler mode and reviewer mode. 2. **Multiple label application:** When a member applies labels using extensions (ML-assisted labeling, Predictive labeling, Data programming, Label error detection, and Search). 3. **External label acceptance or rejection:** When a member accepts or rejects labels from extensions (ML-assisted labeling, Predictive labeling, Data programming, Label error detection, and Search). 4. **Label update:** When a member updates a label. In reviewer mode, this is counted only if the label is not conflicted. 5. **Label deletion:** When a member deletes a label. 6. **Conflict resolution:** When a reviewer accepts or rejects a conflicted label. Changing a conflicted label to another is recorded as two actions: label rejection followed by label application. 7. **Sentence update:** When a member modifies the data. #### Row labeling, document labeling 1. **Single answer submission:** When a member submits answer. 2. **Multiple answer submission:** When a member submits answers using extensions (ML-assisted labeling, Predictive labeling, Data programming, Label error detection, and Search). 3. **External answer acceptance or rejection:** When a member accepts or rejects answers from extensions (ML-assisted labeling, Predictive labeling, Data programming, Label error detection, and Search). 4. **Answer update:** When a member updates an existing, non-conflicting answer. 5. **Conflict resolution:** When a reviewer resolves conflicts in answers. ## Filter
1. **Project:** View all projects or filter by a specific project. 2. **Member:** View all member activity or filter by a specific member. 3. **Date range:** Use UTC time zone to group activities by day. 4. **Show or hide labeling activities:** Toggle to include or exclude detailed [labeling activities](#labeling-activities). ## Download the activity report Click **Download** at the top right of the page to export the activity report in `.csv` format. The exported report includes tracked activities with the following details: 1. **Timestamp:** The date and time when the activity occurred. 2. **User ID:** The unique identifier of the member who performed the activity. 3. **Display name:** The name of the member who performed the activity. 4. **Email:** The email address of the member who performed the activity. 5. **Mode:** The mode in which the activity was performed (labeler mode or reviewer mode). If it's not a labeling activity, it will be categorized as **General**. 6. **Project ID:** The unique identifier of the project where the activity occurred. 7. **Project name:** The name of the project where the activity occurred. 8. **Document ID:** The unique identifier of the document associated with the activity. 9. **Document name:** The name of the document associated with the activity. 10. **Action:** The specific action taken by the member. This includes workspace-level actions (for example, creating a project) and project-level actions (for example, creating a label). 11. **Additional data:** Extra information not captured in other columns, stored in `.json` format. # File Transformer Allows you to write a custom implementation of some features in Datasaur. File transformer feature can be found in **Team workspace**. You can hover over to the left sidebar and click **File Transformer**.
Click **Create file transformer** to start writing your script. Enter a name, then choose the type of file transformer: * Select **Import** to create a script for importing files into a project. * Select **Export** to create a script for exporting files from a project.
### External dependencies Datasaur no longer support importing external dependencies. To make use of common helpers, embed the relevant code snippets directly in your file transformer. # Import Transformer ## Introduction With the **import** transformer, you can import almost any text format into Datasaur. Currently, we limits the file extensions into `.csv`, `.txt` and `.json`. ### Text-based projects Creating a new import transformer will provide you with the following template, which works for **Span labeling**, **Row labeling**, and **Span+Line labeling** projects: {% hint style="info" %} For **Bounding box labeling** projects, see the next section [#bounding-box-labeling-project](#bounding-box-labeling-project "mention") {% endhint %} {% code overflow="wrap" lineNumbers="true" expandable="true" %} ```typescript /** * This function should be written as this template and correctly implements ImportFunction interface. */ (fileContent: string): SimpleDocument => { /// Implement import function here return { cells: [], labels: [], }; }; ``` {% endcode %} The import transformer is a function that takes the `fileContent` as a string, parsed using **UTF-8 encoding**, and returns a `SimpleDocument` that Datasaur can process. `SimpleDocument` is an object that represents a document in Datasaur. It is a combined type that supports span labeling and row labeling. The structure of `SimpleDocument` is shown below:
SimpleDocument structure * **cells:** An array of cells. Datasaur documents are stored in a tabular structure. Each cell represents a single table cell. For span-based projects, only a single-column table is currently supported. Each row in the document must have the same number of columns. * **line:** A zero-based number indicating the row. * **index:** A zero-based number indicating the column. For span-based projects, this value can only be set to `0`. * **content**: The original content of a cell. * **tokens**: A tokenized version of the content. This field is only used for token-based projects only. * **metadata**: An optional array of key-value data to be stored per cell. You can find the structure and configuration options for metadata [here](/advanced/extensions/metadata). * **key**: Identifier for the metadata item, represented as a string. Example: `author`. * **value**: Content or data of the metadata item, represented as a string. Example: `John Doe`. * **type**: Optional field indicating the type of the value in [MIME type](https://developer.mozilla.org/en-US/docs/Web/HTTP/Basics_of_HTTP/MIME_types/Common_types). * **Default**: text/plain. * **Supported type**: * **text/plain:** Displays metadata as plain text. * **text/html**: Displays metadata as HTML. * **image/\*:** Displays metadata as an image. [Supported image formats](https://en.wikipedia.org/wiki/Comparison_of_browser_engines_\(graphics_support\)) depend on your browser. * **audio/\*:** Displays metadata as an audio player. [Supported audio formats](https://en.wikipedia.org/wiki/HTML5_audio#Supported_audio_coding_formats) depend on your browser. * **pinned**: Boolean that specifies whether metadata is shown at the top of each cell. Metadata that isn’t pinned remains available in the **Metadata** extension. * **config**: Customizes appearance for text/plain types. * **color:** Determine the text color of the metadata as a string. Accepts any HTML color codes and names. * **backgroundColor:** Determine the background color of the metadata as a string. Accepts any HTML color codes and names. * **borderColor:** Determine the border color of the metadata as a string. Accepts any HTML color codes and names. * **labels:** An array of labels. * Common fields: * **id:** A unique number to identify the label, to be referred to by the arrow labels. * **startCellLine**: Starting line position. * **startCellIndex**: Starting line column position. * **startTokenIndex**: Starting token index position relative to cell. * **startCharIndex**: Starting character index position relative to the token. * **endCellLine**: Ending line sentence position. * **endCellIndex**: Ending line column position. * **endTokenIndex**: Ending token index position relative to cell. * **endCharIndex**: Ending character index position relative to the token. * **type:** Type of the labels. Must be one of: `"SPAN"`, `"ARROW"`, `"BOUNDING_BOX"`, `"TIMESTAMP"`. * Specific fields by its type: * `"SPAN"` or `"ARROW"` * **labelSetIndex**: Replaces **layer**. Configures how the label set items are grouped. * **labelName**: Replaces **labelSetItemId**. The text provided here will be displayed in web UI. * `"ARROW"` * **originId:** ID of a span label as the arrow's origin. * **destinationId:** ID of a span label as the arrow's destination. * `"BOUNDING_BOX"` * **pageIndex**: Page information for multiple page files, such as `.pdf` and `.tiff`. Set field to **0** for common image formats, such as `.jpg`, `.png`, `.bmp`, etc. * **nodeCount**: Number of nodes; this is used for future support for polygons. Only supports **4** nodes in a rectangular shape for now. * **x0:** The first node's x value in the screen coordinate system. * **y0:** The first node's y value in the screen coordinate system. * **x1:** The second node's x value in the screen coordinate system. * **y1:** The second node's y value in the screen coordinate system. * **x2:** The third node's x value in the screen coordinate system. * **y2:** The third node's y value in the screen coordinate system. * **x3:** The fourth node's x value in the screen coordinate system. * **y3:** The fourth node's y value in the screen coordinate system. * `"TIMESTAMP"` * **startTimestampMillis:** The starting timestamp in milliseconds. * **endTimestampMillis:** The ending timestamp in milliseconds.
#### Sample This example shows how to label a `.srt` subtitle file and display timestamps as metadata. The file transformer script is shown below. {% code title="" lineNumbers="true" expandable="true" %} ```typescript /** * This function should be written as this template and correctly implements ImportFunction interface. */ (fileContent: string): SimpleDocument => { /// Implement import function here const lines = fileContent.split('\r\n\r\n'); let currLine: number = 0; const cells: Cell[] = []; lines.forEach((line) => { const [, timestamp, ...subtitles] = line.split('\r\n'); subtitles.forEach((subtitle) => { cells.push({ index: 0, line: currLine, content: subtitle, tokens: subtitle.split(' '), metadata: [ {key: "timestamp", value: timestamp, pinned: true, config: { color: "#3399cc", backgroundColor: "", borderColor: "#cc3399"}} ] }); currLine += 1; }); }); const labels: SpanAndArrowLabel[] = []; let labelId = 0; // Label the first two tokens on the second line as "Example label" const secondTokenOnSecondLine = cells[1].tokens[1]; labels.push({ id: ++labelId, type: "SPAN", startCellLine: 1, startCellIndex: 0, startTokenIndex: 0, startCharIndex: 0, endCellLine: 1, endCellIndex: 0, endTokenIndex: 1, endCharIndex: secondTokenOnSecondLine.length - 1, labelSetIndex: 0, labelName: "Example label" }) // Label each occurence of "Sherlock" as "Person's name". const sherlock = "sherlock"; cells.forEach(cell => { cell.tokens.forEach((token, tokenIndex) => { if (token.toLowerCase() === sherlock) { labels.push({ id: ++labelId, type: "SPAN", startCellLine: cell.line, startCellIndex: cell.index, startTokenIndex: tokenIndex, startCharIndex: 0, endCellLine: cell.line, endCellIndex: cell.index, endTokenIndex: tokenIndex, endCharIndex: token.length - 1, labelSetIndex: 0, labelName: "Person's name", }) } }) }) return { cells, labels, }; }; ``` {% endcode %} To get started: 1. Rename your file by adding the `.txt` extension. You can use the sample file below. 2. Click **Create file transformer.** 3. Enter a name, select **Import**, then click **Create**. 4. Paste the file transformer script to the editor. You can also upload it.
5. Go to **Projects** page and click **Create a project**. 6. In step 1, select the file transformer you just created in the **File transformer** dropdown. Finish the project creation and launch the project.
Your project is ready!
**Notes:** * You need to add **Metadata** extension to the project. * If you want the metadata to be available in the text editor, set `pinned: true`. * Use HTML code color for text color, border color, and background color. ### Bounding box labeling projects Bounding Box projects support import file transformers when creating a project from a PDF or image. The transformer script receives the paired answer file's content (not the raw PDF/image bytes) as a UTF-8 string, and must return a `BoundingBoxSimpleDocument`. Pairing rule: the answer file must share the same basename as the media file (e.g. `invoice.pdf` + `invoice.json`). If no matching answer file is found, project creation fails with an `ANSWER_FILE_NOT_FOUND` error. {% hint style="info" %} The transformer is always invoked with a bare string, reading page geometry on-the-fly is not yet supported. Bounding box coordinates returned by the script must be in absolute pixels. {% endhint %} Here's a minimal file transformer that will apply a single bounding box to the document regardless of the answer file content: {% code title="" overflow="wrap" lineNumbers="true" expandable="true" %} ```typescript (fileContent: string): BoundingBoxSimpleDocument => { // customize the import function here return { cells: [ { line: 0, index: 0, content: "placeholder", tokens: ["placeholder"] } ], type: "BOUNDING_BOX", labels: [], bboxLabelSets: [ { name: "BBox label set", classes: [ { captionAllowed: true, captionRequired: false, name: "label-01", color: "red" } ] } ], bboxLabels: [ { id: "1", bboxLabelClassId: "label-01", bboxLabelClassName: "label-01", shapes: [ { pageIndex: 0, points: [ { x: 100, y: 100 }, { x: 200, y: 100 }, { x: 200, y: 200 }, { x: 100, y: 200 }, ] } ], } ], }; }; ``` {% endcode %} If you have any questions, please reach out to . # Export Transformer With the **export** transformer, you can export almost anything out of Datasaur.\ \ Your new export transformer will have this template: {% code title="" %} ``` /** * This function should be written as this template and return string. */ (document: Exportable): string => { /// Implement export function here return document.cells.map((cell) => cell.content).join('\n'); }; ``` {% endcode %} ## Sample case This example shows how to export a span labeling in a format compatible with Google AutoMl. The file transformer script is shown below, written in TypeScript: ``` function getCellMap(cells: Cell[]) { const cellMap = new Map(); cells.forEach(cell => { cellMap.set(cell.line, cell); }) return cellMap; } function getLabelSetMap(labelSets: LabelSet[]) { const labelSetMap = new Map(); labelSets.forEach(labelSet => { labelSet.labelItems.forEach(labelItem => { labelSetMap.set(labelItem.id, labelItem); }) }); return labelSetMap; } function convertOffset(label: SimpleLabel, cell: Cell) { const offset = { "end_offset": 0, "start_offset": 0 }; const startTokenIndex = label.startTokenIndex; const endTokenIndex = label.endTokenIndex; const startCharIndex = label.startCharIndex; const endCharIndex = label.endCharIndex; let offsetCounter = 0; for (let i = 0; i <= endTokenIndex; i++) { if (i == startTokenIndex) { offset.start_offset = offsetCounter + startCharIndex; } if (i == endTokenIndex) { offset.end_offset = offsetCounter + endCharIndex + 1; break } offsetCounter = offsetCounter + cell.tokens[i].length + 1; } return offset; } function stringifyWithSpaces(obj) { let result = JSON.stringify(obj, null, 1); // stringify, with line-breaks and indents result = result.replace(/^ +/gm, " "); // remove all but the first space for each line result = result.replace(/\n/g, ""); // remove line-breaks result = result.replace(/{ /g, "{").replace(/ }/g, "}"); // remove spaces between object-braces and first/last props result = result.replace(/\[ /g, "[").replace(/ \]/g, "]"); // remove spaces between array-brackets and first/last items return result; } /** * This function should be written as this template and return string. */ (document: Exportable): string => { /// Implement export function here const cellMap = getCellMap(document.cells); const labelSetMap = getLabelSetMap(document.labelSets); const examplesMap = new Map(); document.labels.forEach(label => { const labelItem = labelSetMap.get(label.labelSetItemId); const cell = cellMap.get(label.startCellLine); const offset = convertOffset(label, cell); const annotation = { "text_extraction": {"text_segment": offset}, "display_name": labelItem.labelName }; if (examplesMap.has(label.startCellLine)) { const example = examplesMap.get(label.startCellLine); example["annotations"].push(annotation); examplesMap.set(label.startCellLine, example); } else { const example = { "annotations": [annotation], "text_snippet": {"content": cell.tokens.join(' ')} }; examplesMap.set(label.startCellLine, example); } }); let output = []; examplesMap.forEach((value) => { output.push(stringifyWithSpaces(value)); }) return output.join('\n'); }; ``` Follow these steps to upload and export using a custom export transformer: 1. Go to the **File transformers** page. 2. Click **Create file transformer.** 3. Enter a name, select **Export** as the purpose, then click **Create.** 4. Paste the file transformer script to the editor. You can also upload it. 5. Go to **Projects** page and open project you want to export. 6. Click **File** > **Export file**. 7. In the **Format** field, select **Custom Format (via File Transformer)**. 8. In the **Export as** field, add the `.jsonl` extension at the end of the file name. 9. In the **File transformer** field, select the file transformer you just created. If you have any questions, please reach out to . # Upload File Transformer To create a file transformer, you can also upload an existing script. Only files with `.js`, `.ts`, and `.txt` extensions are supported. Use the **Upload** button on the right side of the editor to upload the file.
After uploading, the current file transformer will be replaced with the newly updated version.
# Run File Transformer This feature allows you to check whether the file transformer you built is working properly or not. ### Run import file transformer After creating a file transformer, you can test it by following these steps: 1. Upload a file in the **Input** section.
2. Click **Run**.
3. View the result in the **Output** section.
### Run export file transformer After creating an export file transformer, you can test it by following these steps: 1. In the **Input** section, click **Select sample** and choose a sample. You can select a file from an existing project or use a sample template. ![](/files/2lnYD8hZIQNzTNyNOmKk) 2. If you select **File from existing project**, a dialog will appear for file selection. Select a project on the left, then choose one file on the right side. ![](/files/v4V5w8jeg5bTSGawMXfZ) 3. If you don't have any existing projects in the workspace, choose **Sample templates**. A dialog will appear for template selection. ![](/files/R9KPK7UwzAlG6zOXuEwZ) 4. After you select a file, the input will be displayed in JSON format. ![](/files/YFNEOZQJDnEXzcRfFik7) 5. Click **Run**, then view the result in the **Output** section. ![](/files/YDhp1u9drvnqJEf2SJA9) # Label Management In the **Label management** page, you can manage label sets and question sets in your team workspace. To access it, you can hover over the left sidebar and click **Label management**.

Label management

By saving your label sets and question sets on this page, you can reuse them in new projects without creating them from scratch. Learn more about label management here: * [Label Set Management](/workspace-management/label-management/label-set-management) * [Question Set Management](/workspace-management/label-management/question-set-management) # Label Set Management Manage label sets in a team workspace. With label set management, you can save and edit label sets outside a project. You can later reuse them in span labeling or conversational labeling projects. Learn more about the [label set structure](/compatibility-and-updates/common-terminology#labeling-concepts). ## Label set management page {% hint style="info" %} Changes made in the **Label management** page are not automatically applied to existing projects. {% endhint %} ![](/files/4V2JkYOcnwtTge9YW8iT) In the **Label sets** tab, you can add new label sets, edit existing label sets, or delete them. ### Add label set There are two ways to add label sets: upload a label set or create it from scratch. #### Upload a label set You can upload an existing label set by dragging and dropping the file into the upload area. Supported formats: `.csv` , `.json` , and `.tsv` . Refer to the sample format below to ensure compatibility. {% file src="/files/gQ6GKRYXFAirMMAiwlxu" %} {% file src="/files/w4wT371LlXaY95ZX3rdo" %} {% file src="/files/rktoNIvZEMF0VbmfUAkE" %} #### Create label sets from scratch You can also create a label set manually. The dialog includes two tabs: **Standard** and **JSON.** These tabs are synchronized, changes in one tab automatically update the other. To create a label set in the **Standard** tab: 1. Type a label class. 2. Choose a label color by selecting a color from the dropdown. 3. Click the **Add clarification** icon and enter the description to add additional information.
### Edit label sets To edit a label set: 1. Select the label set you want to edit. 2. Click the **Edit** icon on the right side.
3. Update the label items and their properties. You can: 1. Rename a label class by editing its name field. 2. Change the label color using the color dropdown. 3. Edit the clarification using the **Edit clarification** icon.
### Delete label sets To delete a label set: 1\. Select one or multiple label sets. 2\. Click the **Delete** button at the top of the table to delete multiple label sets, or click the delete icon on the right side to delete a single label set. 3\. Confirm the action.
### Search label sets You can search for label sets by entering keywords in the search bar.
## Use existing label sets during project creation To use an existing label set when creating a project: 1. In step 3 of project creation, select **Span labeling** or **Conversational labeling**.
2. Click **Select from library** to use an existing label set from the **Label management** page.
3. After you select a label set, you can edit them. Any changes made here will not affect the original label set in the library.
## Update existing projects with existing label sets After a project is created, you can modify the label set if needed. You can reuse label sets from the **Label management** page. If the **Label set modification** setting is turned off, only admins or project owners can modify label sets. If it is turned on, labelers can also modify them. This setting is turned on by default. For details, visit [question set modification](/advanced/extensions/document-and-row-labeling#question-set-modification). # Question Set Management Manage question sets in a team workspace. With question set management, you can save and edit question sets outside a project. You can later reuse them in row labeling, document labeling, and line labeling projects. ## Label management page {% hint style="info" %} Changes made in the **Label management** page are not automatically applied to existing projects. {% endhint %}
In the **Question sets** tab, you can add new question sets, edit existing question sets, or delete them. ### Add question sets There are two ways to add question sets: upload a question set or create it from scratch. #### Upload a question set You can upload an existing question set by dragging and dropping the file into the upload area. Supported formats: `.csv` , `.json` . Refer to the sample format below to ensure compatibility. {% file src="/files/i1PGcOaGbvbnkBPUo58w" %} #### Create question sets from scratch You can also create a question set manually. The dialog includes two tabs: **Standard** and **JSON.** These tabs are synchronizedchanges in one tab automatically update the other.
### Edit question sets To edit a question set: 1. Select the question set you want to edit. 2. Click the **Edit** icon on the right side.
3. Update the question properties.
### Delete question sets To delete a question set: 1\. Select one or multiple question sets. 2\. Click the **Delete** button at the top of the table to delete multiple question sets, or click the delete icon on the right side to delete a single question set. 3\. Confirm the action.
### Search question sets You can search for question sets by entering keywords in the search bar.
## Use existing label sets during project creation To use an existing label set when creating a project: 1. In step 3 of project creation, select **Row labeling**, **Document labeling**, or **Line labeling**.
2. Click **Use existing question set** to select an existing label set from the **Label management** page. You can preview the question set as well before selecting.
3. After you select a question set, you can edit them. Any changes made here will not affect the original question set in the library.
## Update existing projects with existing question sets After a project is created, you can modify the question set if needed. You can reuse question sets from the **Label management** page. If the **Question set modification** setting is turned off, only admins or project owners can modify question sets. If it is turned on, labelers can also modify them. This setting is turned on by default. For details, visit [question set modification](/advanced/extensions/document-and-row-labeling#question-set-modification). # Project Management ## Project filters In the **Projects** page, you can filter projects by tags, project type, created date, last modified date, status, and assignees.

Filter by tags

Filter by project types

Sort by date created

Sort by date modified

Filter by members

Filter by status

### Project status Filtering projects by labelers and status can help you manage team [team projects](/workspace-management/workspace) more effectively. Refer to the [status workflow](/workspace-management/project-management/project-status-workflow) for more details. ## Manage tags Tags help you organize and quickly find projects, especially when managing many projects with different purposes.
### Create tags You can create tags in two ways: from the **Projects** page or during project creation. #### From the Projects page 1. Click the **three-dot menu** on a project. 2. Select **Manage tags**.
3. Enter a tag in the **Type a tag** field. ![](/files/zrrR1yoQmQ5V00QPzRTM) 4. Click **Save changes.**
#### During project creation To apply tags during project creation, select them in the **Tags** field in step 1.
### Delete tags In the **Manage tags** dialog, hover over a tag and click the delete icon. ![](/files/tQneBxB277P9d1XTSikO) ## Delete projects You can delete projects individually or in bulk. To delete a project individually, click the **three-dot menu** on a project and select **Delete**.
To delete multiple projects, select the projects you want to delete, then click the **Delete** button above the project table, then confirm the action.
## Reuse project settings You can create a new project by reusing settings from an existing one. 1. Click the **three-dot menu** on a project you want to reuse. 2. Select **Reuse project settings**.
You will be redirected to the project creation flow. The new project name includes a `[Copied settings]` suffix, and the configuration will match the selected project.
Only the following settings are reused: * Task type * Task settings * Label sets or question sets * Step 5 settings in the project creation flow * Project extensions used in the original project You still need to upload data and assign labelers and reviewers. {% hint style="info" %} Reusing settings across different project types is not supported. For example, settings from a row labeling project cannot be applied to a span labeling project. {% endhint %} ## Share projects You can share a project link in two ways: * From the three-dot menu on the **Projects** page and click **Share.**
* From the **Share** icon inside the project.
Only assigned members can open the link. For example: * If you share the link with an assigned labeler, they can open it. * If you share it with a reviewer, they can open it in reviewer mode. * If you share it with someone who is not assigned, they cannot open it. # Manage Documents in an Ongoing Project In some cases, you may need to update the documents inside an ongoing labeling project. You can manage project documents without creating a new project. This includes adding new documents and deleting existing ones. To learn which roles can add or delete project documents, see [Role & Permission](/workspace-management/role-and-permission). ## Add new documents To add new documents to an existing project: 1. Go to the **Projects** page. 2. Click the three-dot menu on the project you want to update. 3. Select **Add new documents to the project.** 4. You will be directed to the file upload page.
There are three steps in the file upload page: upload, assign, and preview. ### **Upload files** Upload your new files and make sure the file types match the original project setup. ### **Assign documents** Assign team members to the uploaded documents. Assignments apply to the entire batch of documents, not individual documents. If you need more granular control, you can skip this step and assign documents later in **File** → **Settings** → **Assignment** after the upload is complete. Role options may be limited based on a member’s previous assignment in the project. The table below shows how previously assigned roles map to available options.
#Previous Assigned RoleAvailable Roles
1Not assignedLabeler, Reviewer, Labeler and Reviewer
2LabelerLabeler, Labeler and Reviewer
3ReviewerReviewer, Labeler and Reviewer
4Labeler and ReviewerLabeler and Reviewer
For example, if a member was assigned as a labeler during project creation, they can only be assigned as **Labeler** or **Labeler and Reviewer** for newly uploaded documents. Available roles also depend on the workspace role. For example, if a workspace admin has not been assigned to the project and you try to assign them to newly uploaded document, only **Reviewer** and **Labeler and Reviewer** will be available, since a team admin cannot be assigned as a labeler only. ### **Preview files** You can preview the files you uploaded before completing the process. This lets you check whether the correct files were added. ## Delete documents You can delete documents from an ongoing project in two ways. #### Delete from the Projects page To delete documents from the **Projects** page: 1. Go to the **Projects** page. 2. Open **Manage documents** for the project. 3. Delete documents in one of these ways: * Click the **Delete** button for a specific file to remove that file only. * Use the checkboxes to select multiple files, then delete them in bulk.
#### Delete from inside a project To delete documents from inside a project: 1. Open the project in **Reviewer mode**. 2. Open the **List of files** extension. 3. Delete documents in one of these ways: * Right-click a specific file and select **Delete**.
* Use the checkboxes to select multiple files, then delete them in bulk.
## Additional information * Project status: * Adding or deleting documents does not change the [project status](/workspace-management/project-management/project-status-workflow). * If you assign new documents to users who already completed their work, their labeling status changes back to **In progress**. * Webhook events — Instead of implementing a new webhook event for adding documents, we are reusing the project created event. We are also planning to implement a separate event before rolling this out for general release. # Self-Assignment In some workflows, you may want labelers to pick up work on their own instead of being assigned in advance. With self-assignment, labelers can assign documents to themselves from a pool of available projects. This helps ensure unassigned documents are quickly handled and gives labelers more control over their workload. {% hint style="warning" %} Self-assignment is not enabled by default in projects created before version 6.90. {% endhint %} ## Enable self-assignment Admins can enable self-assignment in workspace settings: 1. Go to the **Workspace settings** page from the left sidebar. 2. Open the **Assignment** tab. 3. Enable **Allow self-assignment for labelers.**
4. Click **Save changes**. Once enabled, there are changes in the **Projects** page depending on the role: * **Admins** and **supervisors** can see projects with incomplete assignments marked with an **Incomplete assignment** chip in the **Status** column.
* **Labelers** will see a new **Unassigned projects** tab, where they can view projects they are not assigned to and self-assign documents.
## Prepare projects for self-assignment To make a project available for self-assignment, create the project as usual. In step 4 of project creation, assign only **reviewers** and leave labeler assignments empty. ![](/files/ln9AxGYOJ8YKjf8simiR) ## Self-assign Once enabled, labelers can self-assign documents from three places: the **Unassigned projects** tab, the **Assigned projects** tab, or within a project. ### Unassigned projects tab Labelers can assign themselves to a project they have not worked on from the **Unassigned projects** tab. 1. Go to the **Projects** page. 2. Open the **Unassigned projects** tab. ![](/files/Hynmr3nv7xAP9OxQHYva) 3. Click **Assign myself** for the desired project. 4. Enter the number of documents and click **Assign to me.**
The project will open automatically after a successful assignment. ### Assigned projects Labelers can also assign themselves additional documents in projects they are already working on from the **Assigned projects** tab. 1. Go to the **Projects** page. 2. Open the **Assigned projects** tab. 3. In the desired project, click the three-dot menu and select **Assign myself to more documents**. ![](/files/Q7kcOBiQdLRLQjB9Iiws) 4. Enter the number of documents to assign and click **Assign to me.**
The project will open automatically after a successful assignment. ### Within the project After marking all assigned documents as complete, labelers may be prompted to self-assign more documents if the project still has incomplete assignments. They can choose to assign more documents or mark the project as done.
{% hint style="info" %} If multiple labelers self-assign at the same time, some documents may already be taken. For example, if 5 documents are requested but some are assigned by others first, only 3 may be allocated. {% endhint %} ### Self-assignment for Admins and Supervisors Admins and supervisors can self-assign to documents as labelers. 1. Go to the **Projects** page. 2. In the desired project, click the three-dot menu and select **Assign myself to this project**. ![](/files/7voo7hAw8Z9X9tIvTwD6) 3. Enter the number of documents to assign and click **Assign to me.**
The project will open automatically after a successful assignment. ## Additional configuration You can configure the **document limit** and **time limit** for self-assignment. You can configure them in two places: * In step 5 of project creation.
* In a project via **File > Settings > Administrator**. Changes to time limits only apply to new assignments. Existing assignments keep the original time limit at the time of assignment.
{% hint style="info" %} These settings are project-level and must be configured for each new project. {% endhint %} ### Document limit The document limit controls how many documents a labeler can self-assign in a project. This helps prevent overloading and ensures fair task distribution. The minimum document limit is **1 document**, and the maximum is **1000 documents**. {% hint style="info" %} Since this setting applies only to self-assignment, admins and supervisors can still assign documents to labelers beyond the limit. {% endhint %} #### Example If a document limit is set to 3: * A labeler can self-assign **up to 3 documents**. * Admins can assign **more than 3 documents** to a labeler. * If labelers already have 2 assigned by an admin, they can only self-assign 1 more. ### Time limit The time limit defines how long a labeler has to complete a document once it has been assigned, either through **self-assignment or by an admin**. It helps ensure timely completion and keeps projects on track. The minimum time limit is **1 minute**, and the maximum is **480 hours and 59 minutes**. Details about the countdown: * Countdown starts immediately after assignment. The remaining time is displayed at the top and in the **List of files** extension.
* The countdown stops when the document is marked as complete. Once completed, the document's status cannot be reverted to **In progress**. * If time expires, the document is unassigned and returned to the available pool. # Self-Unassign You can unassign yourself from a project when needed. This can happen if you are unable to complete the task, need to prioritize other work, or were assigned by mistake. {% hint style="info" %} **Notes** * Self-unassign is available only when self-assignment is enabled. * The project must be in progress. You cannot unassign from completed projects. * If all documents are marked as complete, unassign is disabled to prevent removing yourself from finished work. {% endhint %} You can unassign yourself from documents in the **Assigned projects** tab or within a project. ## Assigned projects tab 1. Open the **Projects** page. 2. Go to the **Assigned projects** tab. 3. Click the **three-dot menu** on a project and select **Unassign incomplete documents**. ![](/files/Q7kcOBiQdLRLQjB9Iiws) If the project contains both labeled and unlabeled documents, select which documents to unassign. If all documents are either fully labeled or unlabeled, click **Unassign documents**.
## Within the project 1. Open the project. 2. Go to **File >** **Unassign incomplete documents.** For admins, supervisors, and reviewers, ensure you are in labeler mode for the unassign menu to appear. ![](/files/XHHKcV66GQGxsgDNnmuK) If the project contains both labeled and unlabeled documents, select which documents to unassign. If all documents are either fully labeled or unlabeled, click **Unassign documents**. ![](/files/xKiy53EZdvTcS5CFb5Aw) # Transfer Assignment Ownership In some cases, a labeler may leave a project or become unavailable before completing their assignment. Instead of adding a new labeler who has to start from scratch, you can transfer assignment ownership so another labeler can continue the work from where it was left off. {% hint style="info" %} This feature is available only in team workspaces and is disabled by default. Reach out to **** if your team needs this feature. {% endhint %} ## Transfer assignment {% hint style="info" %} You cannot transfer an assignment to a member who was previously assigned to the project and then unassigned. {% endhint %} In a project, go to **File > Settings > Assignment** to manage assignments. Only workspace admins can update project assignments. ![](/files/UcseDZ40VPaPUKuE2xH7) If assignment transfer is enabled, an additional column, **Transfer work to**, appears. * Only labeler work can be transferred. * Eligible roles: **Labeler** and **Labeler & Reviewer**. * Reviewer assignments do not require transfer. Reviewers share the same work, so assigning a new reviewer allows them to continue progress immediately. ## Limitation Transferring work may affect analytics, especially inter-annotator agreement ([IAA](/workspace-management/analytics/inter-annotator-agreement)). Label ownership is derived from document ownership, so after a transfer, labels created by the previous labeler are attributed to the new labeler. An info icon appears next to the labeler name to indicate that a transfer has occurred.
This can affect IAA results because agreement is calculated between labelers. When labelers do not work on the same document from start to finish due to transfers, the results may be skewed. # Reset Labeling Work ### Overview The **reset labeling work** option allows a labeler to restart labeling from scratch by clearing all labels in the currently active document. {% hint style="info" %} * This option is available only in **labeler mode**. * This action is **permanent** and **cannot be undone.** * In mixed labeling projects, resetting will remove all labeling types within the project, not just a specific type. {% endhint %} ### Reset labeling work 1. In a project, go to **File** > **Reset labeling work in this document**. ![](/files/6NAc87JOMaN6EjXE5QpK) 2. A confirmation dialog will appear. Click **Reset** to continue. The reset process may take a few moments. You’ll receive a notification once it's complete. If some labels still appear, refresh the page to ensure all previous labels are cleared. # Mark Document as Complete Marking a document as complete helps you track your progress and identify the last labeled document. It also helps distinguish finished documents when working with multiple files in a project. ## Mark a document as complete 1. Open a project that contains multiple documents. 2. Open the **List of files** extension. 3. Click the document icon next to a document name to mark it as complete.
In span labeling projects, if some label classes are not applied in the document, a confirmation dialog will appear indicating the missing labels.
You can also mark a document or project as complete from the **File** menu by selecting: * **Mark document as complete** * **Mark project as complete** A keyboard shortcut is also available.
If all documents in a project are marked as complete, a confirmation dialog will appear prompting you to mark the project as complete. ![](/files/s1Oh2Ae6Ux8YDvSDGNUs) # Project Status Workflow The project status helps you track which projects are complete and which are not. You can view the project status in the **Projects** page and filter projects by status. ![](/files/VsqyooCeJNeNeXaCTR5w) There are 2 project statuses for each project in the **Projects** page: 1. **Overall status:** Reflects the project’s latest status, regardless of your role. There are 5 possible statuses: 1. **Created** 2. **In progress** 3. **Ready for review** 4. **In review** 5. **Completed** 2. **Your labeling status:** Reflects your labeling progress. This appears only if you are assigned as a labeler in the project. There are 3 possible statuses: 1. **Not started** 2. **In progress** 3. **Completed** The images below show all possible project status flows. ![Project status state diagram](/files/-MbjYHkn8q1RB44AwEnS) # Read-only Mode Read-only mode is a state where you can view the project after it is marked as complete, but cannot make any changes. When labelers mark a project as complete, they can view it in read-only mode.\ When reviewers mark the project as complete, they can also view it in read-only mode, and all assigned labelers receive an email notification informing them that the project has been reviewed. {% hint style="info" %} In read-only mode, you cannot label data or enable extensions. {% endhint %}
After the review is finished, labelers can review the project in read-only mode as feedback to improve their labeling performance.
Reviewers can also access the project in read-only mode to view work from other labelers.

Show work from all labelers

Show work from selected labeler

Reviewers and labelers can view labeling statistics in the status bar at the bottom right.
# Comment Feature Comment is available for both span and row-based projects with tabular view. Comments let you communicate directly within a project while labeling or reviewing. You can use them to collaborate, ask questions, provide feedback, or highlight specific parts of a document without affecting labels. This option is available in both personal and team workspaces. ### Comment on a label To leave a comment: 1. Highlight the label you want to comment on. 2. Right-click that highlighted label and select **Comment**. ![](/files/TkGFsYxCcRDsfjPt3OvM) 3. Enter your comment.
If a label already has comments, you can open the thread to view previous messages. You can also access all comments from the **Comments** extension without clicking each label. ### Edit a comment To edit a comment: 1. Click the three-dot menu on a comment and select **Edit**.
2. Update the comment and click **Save**.
### Resolve a comment Click the check icon on a comment to resolve that comment. Both labelers and reviewers can resolve comments.
Once resolved, the comment will be moved to the **Resolved** tab. It will disappear from the text editor and can only be accessed from the **Comments** extension in the **Resolved** tab.
### Reopen a comment To reopen a resolved comment: 1. Go to the **Resolved** tab. 2. Click the three-dot menu on a comment and select **Re-open**.
### Delete a comment To delete a comment: 1. Open the three-dot menu. 2. Select **Delete**. Only admins or the comment creator can perform this action.
### Comment on a span To comment on a span of tokens: 1. Highlight the text. 2. Right-click the selection. 3. Select **Comment**.
### Comment in a row labeling project To comment in a row labeling project with tabular view: 1. Right-click the cell you want to comment on. 2. Select **Comment**. You can also comment on empty cells to coordinate with your team.
# Archive Project This feature allows you to archive old projects and keep your board clean. ## Archive projects You can archive projects if you no longer need to work on them. Archived projects are removed from your active list but remain accessible for future reference. They are still editable as well. ### Archive one project To archive a project, click on the triple-dots menu then select **Archive**. ![](/files/mswQV3yGohuPWg1eEyDe) ### Archive multiple projects To archive multiple projects, select the checkboxes on the left, then click **Archive** at the top of the table. ![](/files/1LGTY3XNWWxqiNqxANXj) A confirmation dialog will appear. Click **Archive projects** to confirm. ![](/files/yiCRyVRGPlJ5Ngnqf3RC) ## View archive projects To view archived projects, go to the left sidebar, open the **Projects** dropdown, then select **Archived**. ![](/files/ngOtk2lFdiwDZyLtxZew) You will see a list of all archived projects. ![](/files/mRHdX3f9TczVsSWbFcrP) ## Restore archive projects ### Restore one project To restore a project, click on the triple-dots menu then select **Restore**. ![](/files/WczbH8y5U4DjtE4lhysL) ### Restore multiple projects To restore multiple projects, select the checkboxes on the left, then click **Restore** at the top of the table. ![](/files/PiXYccta3QbTjjf9L3pW) # Automation Using Datasaur Action allows you to automate certain processes in the app. Actions help automate repetitive processes across your workflow. Actions bring bulk action capabilities from [Robosaur](/integrations/robosaur) (for example, creating or exporting multiple projects) into the Datasaur app, so you can manage and automate tasks in one place. Currently supported actions: * [Create projects](/workspace-management/automation/action-create-projects) # Action: Create Projects Automate project creation using files from your external object storage. This action relies on two required components: * [External object storage](/integrations/external-object-storage): The system accesses files from a specified path in your bucket. * [Project template](/data-studio-projects/nlp-task-types/project-templates#create-your-own-template): The system uses a project template that defines the configuration that each created project will follow. ### Create or edit an action * To create, click the **New action** button on the page. * To edit, click the three-dot menu on an action, then click **Edit.** #### Step 1
All fields are required: 1. Name your action. 2. **Project template:** The projects that will be created by the action will follow this configuration. You must create a template before setting up the action. 3. **External object storage:** The action will fetch data from this bucket for the projects. You must set up your object storage before creating an action. 4. **Input path:** The system will read files from this path when the action runs. Each folder represents a project, and each file inside it becomes a document. 1. This path is relative to the root of your bucket. So, if you want to process `S3://test-external-object-storage/input`, you will only need to fill `input` as the value. 2. If multiple folders exist under `input`, the system creates one project per folder. From the example below, two projects will be created: * **Project 1** → `ipsum.txt`, `lorem.txt` * **Project 2** → `lipsum.txt`
5. **Result path:** After the projects are created, the system moves processed files to this path. This prevents files from being processed again in future runs. Do not move files in this result path, as this may prevent the system from accessing them properly. If file conversion is applied (for example, `.docx` → `.pdf`), converted files are also stored here. 6. **Tags** **(optional)**: Add custom tags to projects created by this action. #### Step 2
1. **Assignees**. Select labelers and reviewers to assign to projects created by this action. You must select at least one labeler and one reviewer. 2. **Number of labelers per project**. Defines how many labelers are assigned to each project. If this number is lower than the total selected labelers, assignments are distributed using round-robin. 3. **Number of reviewers per project**. Same as above, but for reviewers. #### Step 3
1. **Number of labelers per document**: Defines how many labelers are assigned to each document. 2. **Conflict resolution**: See [detailed documentation](/data-studio-projects/creating-a-project/consensus) for how conflicts are handled. #### Step 4 Review all settings in one place. You can also see a preview using sample data, including how assignments will be distributed when the action runs. ### Run actions #### From the Datasaur app 1. Go to the **Actions** page. 2. Click **Run** on a specific action. To manage an action (delete or update), use the three-dot menu on the card. 3. To view the activities of an action, click **View run**. 1. Each run appears as one **Action run** (one row), showing details of the automation process.\ You can also see the previous runs. 2. One run can create multiple projects. Click **View details** for breakdowns. 3. If the **View details** button is unavailable, then there is no new data found in the bucket. No projects are created in this case. 4. **Action run details** show each project creation attempt. Each row represents one attempt. #### Through API To do an API call, start from [here](broken://spaces/-MbjXHclk9fELydcAslG/pages/HqlHi3OG0tUpPfWWDtF4). * Mutation: [runAction](https://api-docs.datasaur.ai/#mutation-runAction). # ML-Assisted Labeling ML-assisted labeling extension lets you use open-source models, LLMs, or your own models to generate labels automatically. ## Introduction ML-assisted labeling helps automate data labeling for NLP projects. It supports span, row, bounding box, and document labeling by using open-source models, large language models (LLMs), and custom models. This reduces manual effort and improves labeling speed and consistency.
Service provider for ML-assisted Labeling
## Key features 1. **Batch labeling:** Label multiple items at once, eliminating the need to label each item individually. 2. **Model integrations**: Works with models for tasks such as named entity recognition (NER), part-of-speech (POS) tagging, and sentiment analysis, as well as LLMs and external providers. 3. **Automation:** Generate labels automatically and review them to ensure quality. ## Quick start guide To enable the **ML-assisted labeling** extension: 1. Go to the **Manage extensions** dialog and enable the **ML-assisted labeling** extension.
Image of ML Assisted Labeling Menu

ML-assisted labeling extension with spaCy

2. Select a service provider. 3. Click **Predict labels** to generate labels. {% hint style="info" %} For row labeling projects, there are some additional steps: 1. **Select rows**: Choose which rows to include in prediction. 2. **Target text**: Select input column used as context. 3. **Target question**: Select output field to predict. 4. **Faster prediction speed**: Run predictions via the backend. {% endhint %} ## Supported model providers
Row labelingSpan labelingBounding box labelingDocument labeling
Sentiment AnalysisSpaCyCustom ModelDatasaur LLM Labs
LLM Assisted LabelingLLM Assisted Labeling
Amazon ComprehendNLTK
Google Vertex AICoreNLP and SparkNLP NER
Amazon SageMakerCoreNLP and SparkNLP POS
Datasaur LLM LabsDatasaur LLM Labs
AzureFewNERD
Hugging FaceHugging Face
Custom ModelCustom Model
Model providers are grouped into the following categories: * **Datasaur hosted** * Prebuilt models hosted by Datasaur for common NLP tasks such as NER, sentiment analysis, POS tagging, and dependency parsing. * Examples: spaCy, CoreNLP, and FewNERD. * **Cloud providers** * Models hosted on external platforms. You can use pre-trained or fine-tuned models via API. * Examples: Hugging Face Inference API, Azure ML, Google Vertex AI, and Amazon SageMaker. * **LLM Assisted Labeling** * Models from LLM providers that require an API key. * Examples: OpenAI (GPT models), Azure OpenAI, Anthropic (Claude models), Gemini (Google AI), Cohere, and other custom models that can be connected via API. * **LLM Labs** * Models deployed through Datasaur LLM Labs, providing access to multiple providers through a single endpoint. * This allows users to switch between different models without manually configuring each provider separately. * **Custom models** * Connect your own model using a custom REST API. The API must follow the required request format. * It provides flexibility for organizations with internally trained models or self-hosted LLMs.
Tree-diagram image of ML-assisted Labeling provider in Datasaur

ML-assisted labeling providers in Datasaur

TypeExamples
Datasaur hostedspaCy, CoreNLP, SparkNLP, NLTK, Sentiment Analysis, FewNERD
Cloud providerHugging Face, Azure ML, Google Vertex AI, Amazon SageMaker
LLM Assisted LabelingOpenAI, Azure OpenAI, Anthropic, Gemini, Cohere, custom
LLM Labs100+ LLM providers
CustomDepends on your internal API
## Restrict ML-assisted labeling settings Admins or reviewers can restrict ML-assisted labeling settings to ensure consistent configuration across labelers. When enabled, labelers use the configuration set by the admin or reviewer, ensuring consistent results. ### Steps 1. Click the three-dot menu next to the **ML-assisted labeling** header. 2. In **Modify service provider setting**, choose one of the following options: 1. **All assignees**: Allows all labelers to modify their own settings. 2. **Admin or reviewer only**: Restricts changes to admins or reviewers.
Image of Enabling Admin or Reviewer ML Assisted Labeling Settings to Labeler
### Behaviors * When **Admin or reviewer only** is selected, labelers cannot change the service provider or settings. Admins and reviewers can still update the configuration. * In ongoing projects, labelers must refresh the page to apply updated settings. --- [Next Page](/llms-full.txt/1)