> For the complete documentation index, see [llms.txt](https://docs.datasaur.ai/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.datasaur.ai/agent/labeling-agent.md). # Labeling Agent Labeling agents allow you to assign ML models as labelers in your project and evaluate their performance alongside human labelers. This helps you understand which labeling approach works best for your needs, whether human, machine, or both. Labeling agents simplify the process of testing and comparing ML models inside Datasaur: * You no longer need to create separate accounts or log in as the model to run predictions. * Model outputs are now part of the same analytics and comparison tools used for human labelers. * It’s easier to measure performance and decide what labeling strategy to use. {% hint style="info" %} Models must be deployed in [**LLM Labs**](/llm-projects/deployment.md) in the same team workspace as the Data Studio project. {% endhint %} ## Create a labeling agent {% hint style="info" %} **Supported labeling types**: Span labeling, row labeling. {% endhint %} #### 1. Prepare the label set Make sure the label set you will be using matches the one you configure in step 2. Below is a simple example of labels that can later be used in Data Studio: {% code overflow="wrap" %} ```json { "name": "Labeling agent Label set", "options": [ { "id": "NhsjWIgaAQH3g6dsvtW6a", "color": "#f93b90", "parentId": null, "label": "PERSON" }, { "id": "X1bKK7Nxf9SGaBfDpzH7g", "color": "#d4e455", "parentId": null, "label": "DATE" }, { "id": "NP2RJr7tD5aMfVBnG6TOm", "color": "#85c98e", "parentId": null, "label": "ORG" } ] } ``` {% endcode %} #### 2. Define your instructions In LLM Labs, create a new sandbox and set up the model to act as a labeling agent. To help the model understand what to label, you’ll need to provide clear system and user instructions. The output of the model must be in JSON object format, aligned with the label set defined in your NLP project, to ensure compatibility with regex-based string matching for labeling in your NLP platform. Below is an example setup: **System instruction** ``` You are an expert data labeler ``` **User instruction** {% code overflow="wrap" %} ```` Given the document text, please extract the following information and present it in JSON format as shown below: PERSON: People, including fictional. DATE: Absolute or relative dates or periods. ORG: Companies, agencies, institutions, etc. Instructions Summary: 1. Extract and present the information in the specified JSON format. 2. Ensure that all extracted data is accurate and corresponds directly to the content of each document. Return the value of extracted fields in JSON structure in plain text, following this JSON FORMAT { "PERSON": ["People, including fictional."], "DATE": ["Absolute or relative dates or periods."], "ORG": ["Companies, agencies, institutions, etc."], } VERY IMPORTANT RETURN THE ANSWER WITHOUT ```json ANSWER PRECISELY GIVEN FROM THE SENTENCE PROMPT AND DON'T MASK THE ANSWER, ANSWER BASED ON THE GIVEN SENTENCE ```` {% endcode %} #### 3. Test with a prompt example To check if your instructions work as expected, you can test them using an example sentence. Here's how you might write a prompt: {% code overflow="wrap" %} ``` Label set: - PERSON - DATE - ORG Sentence: Ivan Lee is the CEO and Founder of Datasaur.ai. He graduated with a Computer Science B.S. from Stanford University. He was chosen for the selective Mayfield Fellows entrepreneurship program in 2010. Ivan went on to found Loki Studios, an iOS game studio. After raising institutional funding from DCM's A-Fund and launching a profitable game, Loki was acquired by Yahoo. ``` {% endcode %} After you click the **Run** button, the expected output will be: {% code overflow="wrap" %} ``` { "PERSON": ["Ivan Lee"], "DATE": ["2010"], "ORG": ["Datasaur.ai", "Stanford University", "Mayfield Fellows", "Loki Studios", "DCM's A-Fund", "Yahoo"] } ``` {% endcode %} #### 4. Deploy the model You need to deploy the model first before it becomes available and visible in Data Studio as a labeling agent.
## Assign labeling agents Once you’ve set up the model, you can now assign it as a labeler in Data Studio. #### 1. Assign models as labelers You can assign models during the project creation process: 1. Go to **Projects** page and click **Create new project**. 2. Upload files and select S**pan Labeling** or **Row Labeling**. 3. Configure Labeler’s tasks, for **Span Labeling** you can also enable *Allow arrows to be drawn between labels* checkbox if needed. 4. In the **Assignment** step, open the **Labeling agents** tab. 5. Select the deployed models you want to assign as labeling agents.
Selecting deployed LLM Labs Sandbox as labeling agent

Selecting deployed LLM Labs Sandbox as labeling agent

6. You need to set the agent task by clicking **Set a default agent task** or **Set task scope** on individual labeling agent.
**Configuring labeling agents tasks:**\ \ For a Span Labeling project:

Configuring labeling agent tasks - Span Labeling

1. **Label set:** the label set you want the labeling agent to use in the project. 2. **Allow the labeling agent to apply arrow:** this option is only available if you enable *Allow arrows to be drawn between labels* checkbox in Labeler’s tasks step. Once enabled, you can select the label set for the arrow labeling task.
For a Row Labeling project:

Configuring labeling agent tasks - Row Labeling

1. **Target question**: the question you want the labeling agent to answer. 2. **Input columns:** the columns the labeling agent should use as context. The labeling agent supports the following question types in row labeling projects:
Question typeDescription
RadioSingle-select from a predefined list of options
DropdownSingle-select via a dropdown menu
Hierarchical dropdownNested dropdown with parent-child option relationships
TextFree-form text input
DateDate picker input
TimeTime picker input
CheckboxMulti-select from a list of options
SliderNumeric value selection via a slider control
URLText input validated as a URL
7. Complete the project setup. {% hint style="info" %} You can assign both human members and labeling agents. Each model counts toward your assignment limit. {% endhint %} #### 2. Launch the project and trigger labeling When you click Launch project, the labeling agents will automatically start labeling. **Current limitation:** * Each span will only have one label. * Limited supported question type for optimized performance. #### 3. Review labels applied by the labeling agent Once all documents are fully labeled, either through external model assistance or manual input, the project can undergo a final review. This stage typically involves a reviewer ensuring the consistency and accuracy of all annotations before submission or export through reviewer mode. #### 4. View and compare performance You can track the performance of both human labelers and labeling agents from the [Analytics page.](/workspace-management/analytics.md) From here, you’ll be able to compare IAA scores and other metrics across all labelers, human and models. ## Best practices * Use the external model as a timesaving aid but always include a human review step. * Train your model with high-quality data to improve suggestion accuracy. * Communicate clearly with labelers about how to handle model predictions. * Automate some of the work with consensus by using multiple models, e.g., use the consensus of 3 and deploy 3 Labeling Agents, then focus only on those that are not accepted through consensus. ## FAQs * **Can I assign multiple models to the same project?** * Yes. You can assign up to 10 labeling agents. * **Can I use labeling agents in line labeling project?** * Not yet. They can be assigned to span + line labeling project but will only apply labels for span labeling task. * **How are labeling agent labels shown in the text editor?** * They are treated like human labelers but are masked. You’ll see their labels in the reviewer mode and analytics. --- # Agent Instructions This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com. ## Querying This Documentation If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question. Perform an HTTP GET request on the current page URL with the `ask` query parameter: ``` GET https://docs.datasaur.ai/agent/labeling-agent.md?ask= ``` The question should be specific, self-contained, and written in natural language. The response will contain a direct answer to the question and relevant excerpts and sources from the documentation. Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.