Span Labeling Agent
Use a span labeling agent when you want a model to automatically apply labels to Span Based project type. This is useful for tasks such as named entity recognition, information extraction, and other text annotation workflows where the model needs to highlight and classify specific text spans.
A span labeling agent works alongside human labelers, so you can compare model output, review quality, and measure performance in the same project.
Models must be deployed in LLM Labs in the same team workspace as the Data Studio project.
Before you begin
Before assigning a span labeling agent, make sure you have:
a deployed model in LLM Labs
a span labeling project in Data Studio
a label set prepared for the project
prompt instructions that clearly define what spans the model should label
How span labeling agents work
For span labeling projects, the labeling agent reads the input text and returns predicted spans based on the label set you configure. The output is then applied in the project as model-generated annotations.
A typical setup flow looks like this:
Pre-requisite: Deployed model in LLM Labs
Assign the deployed model as a labeling agent in Data Studio
Launch the project and review the generated labels
Assign the model as a span labeling agent
To assign a deployed model as a labeling agent:
Go to Projects and click Create new project.
Upload your files and Select Span Labeling.
Configure Labeler’s tasks, you can also enable Allow arrows to be drawn between labels checkbox if needed.
In the Assignment step, open the Labeling agents tab.
Select the deployed models you want to assign as labeling agents.
Selecting deployed LLM Labs Sandbox as labeling agent You need to set the agent task by clicking Set a default agent task or Set task scope on individual labeling agent, and complete the project setup.
Configure span labeling tasks
You can configure:
Label set: the label set the agent should use.
Allow the labeling agent to apply arrow: available only if Allow arrows to be drawn between labels is enabled during project setup.
Allow the labeling agent to apply arrow: this option is only available if you enable Allow arrows to be drawn between labels checkbox in Labeler’s tasks step. Once enabled, you can select the label set for the arrow labeling task.
You can assign both human members and labeling agents. Each model counts toward your assignment limit.
Launch the project and trigger labeling
When you click Launch project, the span labeling agent automatically starts labeling the assigned data.
The generated labels can then be reviewed in the project just like other annotations.
Review labels applied by the labeling agent
Once the project has been labeled, reviewers can inspect the annotations to verify:
whether the correct text spans were selected
whether each span was assigned the correct label
whether model output is consistent across similar examples
This review step is recommended before submission, export, or downstream use.
View and compare performance
You can track the performance of both human labelers and labeling agents from the Analytics page.
From there, you can compare IAA scores and other quality metrics across all labelers, including both humans and models.
Limitations
Each span can only have one label.
Model performance depends heavily on prompt quality and label definition clarity.
Last updated