Test Project
Set a test for your labelers!
Last updated
Set a test for your labelers!
Last updated
A Test Project enables you to create a test for your labelers, which can be useful to assess how well they understand the project's guidelines.
The key concept in a Test Project is that all pre-applied labels or labels applied by reviewers are considered as Ground Truth Labels. To pass the test, labelers must ensure that their labels match the Ground Truth Labels.
After assigning the project to the appropriate labelers, click Launch Project.
As the project creator, you will begin in Reviewer Mode by default after launching the project. If you have uploaded an unlabeled file, you can start labeling the spans, which will become the Ground Truth Labels. Uploading a pre-labeled file will save you time.
Once the labelers click Mark as Complete, please navigate to or refresh the Reviewer Mode page. You will see conflicts when the labelers' labels and the Ground Truth labels do not match.
The color-coding convention for All Labelers View is as follows:
Gray: You upload a prelabeled file in TEST project
Violet: all the Ground Truth (GT) labels, i.e.: the labels applied by the reviewer on review mode
Red: all the labels applied by the labelers.
You can see labelers' work by clicking on the dropdown in the Test Project extension.
The color-coding convention for Individual Labelers View is as follows:
Gray: all the labels applied by the labelers which match the GT labels.
Dark Blue: all the labels applied by the labelers which do not match the GT labels.
Violet: all the GT labels are not answered by labelers.
To help assess the labelers' performance, you can see the detailed information about matched labels, incorrect labels, and total labeling time in the extension. These calculations can be downloaded by clicking the triple dots and selecting Export Summary in csv.
All types! You can create span-based, row-based, and document-based projects.
Span-based
Span-based with arrows
Span-based with character based labeling
Row-based
Document-based
The Test Project template is only available on team project, and has 4 steps. Any pre-labeled file conforming to Datasaur's can be uploaded in step 1. All settings in Step 5, as they are not applicable.
Besides uploading unlabeled files, a Test Project allows you to upload pre-labeled files. You can upload files with the format below.
, , ,
, ,
, .,
, .,
