Test Project

Set a test for your labelers!

A Test Project enables you to create a test for your labelers, which can be useful to assess how well they understand the project's guidelines.

The key concept in a Test Project is that all pre-applied labels or labels applied by reviewers are considered as Ground Truth Labels. To pass the test, labelers must ensure that their labels match the Ground Truth Labels.

Create a Test Project

The Test Project template is only available on team project, and has 4 steps. Any pre-labeled file conforming to Datasaur's supported formats can be uploaded in step 1. All settings in Step 5, as they are not applicable.

Assign the appropriate labeler

After assigning the project to the appropriate labelers, click Launch Project.

Start labeling the spans

As the project creator, you will begin in Reviewer Mode by default after launching the project. If you have uploaded an unlabeled file, you can start labeling the spans, which will become the Ground Truth Labels. Uploading a pre-labeled file will save you time.

Reviewer Mode

If you upload a prelabeled file for TEST project, the labels applied will not be displayed in labelers account.

Labeler has finished the project

Once the labelers click Mark as Complete, please navigate to or refresh the Reviewer Mode page. You will see conflicts when the labelers' labels and the Ground Truth labels do not match.

Reviewer Mode with All Labelers View

The color-coding convention for All Labelers View is as follows:

1. Gray: You upload a prelabeled file in TEST project

2. Violet: all the Ground Truth (GT) labels, i.e.: the labels applied by the reviewer on review mode

3. Red: all the labels applied by the labelers.

You can see labelers' work by clicking on the dropdown in the Test Project extension.

Reviewer Mode with Individual Labelers View

The color-coding convention for Individual Labelers View is as follows:

1. Gray: all the labels applied by the labelers which match the GT labels.

2. Dark Blue: all the labels applied by the labelers which do not match the GT labels.

3. Violet: all the GT labels are not answered by labelers.

To help assess the labelers' performance, you can see the detailed information about matched labels, incorrect labels, and total labeling time in the extension. These calculations can be downloaded by clicking the triple dots and selecting Export Summary in csv.

200B

Datasaur sample - Test (summary).csv

What types of projects does Test Project support?

All types! You can create span-based, row-based, and document-based projects.

Upload Pre-labeled Files

Besides uploading unlabeled files, a Test Project allows you to upload pre-labeled files. You can upload pre-labeled files with the format below.

Task type	Formats