Mixed Labeling
This feature allows you to combine two labeling types.
Last updated
This feature allows you to combine two labeling types.
Last updated
Mixed label sets is a feature that allows you to combine two labeling types: span-based labeling and document classification.
Datasaur is currently support:
token-based and document-based types in one interface.
bounding box-based and document-based types in one interface
More combinations coming soon!
Let's follow the steps below to create a span and document-based project!
As an admin, you can create a mixed label set by selecting multiple labeling types, below are the steps to create it.
Open team workspace.
Create a new project.
Upload file(s).
In step 3, you can simultaneously select span labeling and document labeling.
Continue to the next step and launch your project.
Here is the project page view when the mixed label set project is finished.
Once your mixed label set project has been created, you will have an interface looks like below: a label set, and a question set on the right side.
We don’t have any particular rule for doing which type of labeling first. However, we recommend that you label the spans first, then classify the document by answering the question.
If you have multiple documents within a project, submitting the answer will help go directly to the next document.
We use dropdown question type for the sample screenshot above. Please kindly refer to this page for other question types.
Conflict resolution is practically similar with our current behavior. Users will have an information which conflict type in the document (labels, sentence editing, or relation) and the conflict counter in each document.
The difference is, that you will need to display the Document Labeling extension by scrolling down the conflict extension and clicking Document Labeling.
Document Labeling extension will show, and you can resolve the conflicts as resolving conflicts in the row-based or document-based projects.
After the reviewing process is done, you can click Mark as Complete button to indicate that the project is completed!
If there are unresolved conflicts in certain documents, a popup modal shows where the conflicts are. You can open the dropdown, then click the document name. It will go directly to the desired document.
Please note that marking the project as complete will make the project as a read-only. You will need to mark the project as in progress to do some modification.
Here is the way to access Project Analytics.
Project page.
On Project row, press the three dots icon ( ⋮ ) to show options, then select “View Project Analytics”.
Project Analytics Interface.
For more detailed information on Analytics, please refer to this page.
As the Admin, you can download every widget based on the filter applied (date and project type).
These widgets are affected by the project type filter:
Number of tasks
If filtered by document-based, it will be calculated based on document labels.
If filtered by span-based, it will be calculated based on span labels.
Total conflict per labeler
If filtered by document-based, it will be calculated based on document labels.
If filtered by span-based, it will be calculated based on span labels.
Conflict resolved by reviewer
If filtered by document-based, it will be calculated based on document labels.
If filtered by span-based, it will be calculated based on span labels
The IAA on project analytics page will not be filtered by project type.
Here is the way to access Member Analytics:
Members page.
On Member row, press the three dots icon ( ⋮ ) to show options and click “View member details”.
For double typed project, the “labels applied”, “conflicts”, “#of accepted”, and “#of rejected” will be calculated from both project types.
The ML Assisted Labeling will automatically apply to span-based project.
You can use the Dictionary on mixed label set project in the same way as you would use in span-based project.
You can use the Search Extension on mixed label set project in the same way as you would use in span-based project.
You can use the Grammar Checker on mixed label set project in the same way as you would use in span-based project.
The Status Bar provides information that was available on the Dashboard Extension:
Total Labels Applied:
Number of the labels applied on the current file.
Last Labeled Row:
The last labeled row that the labeler/reviewer applied.
Total Labeled Row:
Number of the rows in a File that are labeled by the labeler/reviewer.
The status bar only shows token labels analytics.
Let's follow the steps below to create a bounding box and document-based project!
As an admin, you can create a mixed label set by selecting multiple labeling types, below are the steps to create it.
Open team workspace.
Create a new project.
Upload file(s).
When uploading files, you also have the option to upload pre-labeled files as well.
In step 3, you can simultaneously select bounding box labeling and document labeling.
Continue to the next step and launch your project.
Here is the project page view when the mixed label set project is finished.
Once your mixed label set project has been created, you will have an interface looks like below: a bounding box label set, and a question set on the right side.
We don’t have any particular rule for doing which type of labeling first. However, we recommend that you label the bounding box first, then classify the document by answering the question.
If you have multiple documents within a project, submitting the answer will help go directly to the next document.
Extensions are the same with the bounding box labeling project with additional extension Document Labeling. More information about extensions can be found here.
Following bounding box labeling rules, we currently don't support conflict resolution. It's one consensus settings for now.
Bounding Box-based + Document-based
Document-based only
, ,