Test Project
Set a test for your labelers!
Lets create a test project for your labelers to assess how well they understand the project guidelines.
In a Test project, all pre-applied labels or labels applied by reviewers are considered as ground truth tabels. To pass the test, labelers must match their labels to these ground truth labels.
Create a test project
The Test project template is only available for team projects and has 4 steps. Any pre-labeled file conforming to the supported formats can be uploaded in step 1 of project creation.
Assign the appropriate labeler
Assign the project to the labelers, then click Next.
Start labeling the spans
As the project creator, you will start in reviewer mode by default.
If you upload an unlabeled file, you can label spans yourself; these labels become the ground truth labels.
If you upload a pre-labeled file, you save time, as the ground truth labels are already applied.
If you upload a pre-labeled file for the Test project, the labels will not be displayed in the labeler mode.
Labeler has finished the project
Once the labelers click Mark as complete, navigate to the reviewer mode. Any conflicts between the labelers' labels and the ground truth labels will be visible.
By default, you will see all labelers' work. You can view each labeler’s work using the Show work from dropdown in the Test project extension.
When viewing all labelers’ work, the color coding is:
Gray: Pre-labeled labels.
Violet: Ground truth labels applied by the reviewer on reviewer mode.
Red: Labels applied by the labelers.
When viewing an individual labeler's work, the color coding is:
Gray: Labels applied by the labeler that match the ground truth labels.
Dark blue: Labels applied by the labeler that do not match the ground truth labels.
Violet: Ground truth labels that were not answered by the labeler.
To assess labelers' performance, go to the Test project extension to see the detailed information about matched labels, incorrect labels, and total labeling time. You can download this information by clicking the triple-dot menu and selecting Export summary (.csv).
Supported labeling types
A Test project is available for span labeling, row labeling, and document labeling projects.
Upload pre-labeled files
In a Test project, you can upload pre-labeled files as well as unlabeled files. You can upload pre-labeled files with the formats listed below.
Span labeling
Span labeling with arrows
Span with character-based labeling
Last updated