Analytics

Analytics to track labeler performance and progress in a project.

Analytics

Analytics extension provides real-time performance metrics for each labeler in a project. You can use it to:

• View labeler performance and status, including accepted and rejected labels or answers.

• View how labels and answers are distributed across your label sets and questions.

• Filter by document to analyze performance for specific files.

• Filter by member to compare individual labelers and identify top performers or those who may need support.

Reviewers can view performance data for all assigned labelers even before the project is marked as complete. This helps reviewers track progress throughout the labeling process.

Labelers can only view their own performance after the project has been marked as complete by a reviewer.

The extension has two tabs: Overview and Distribution.

Overview tab

The Overview tab provides a summary of labeling performance and progress in the project. It shows key metrics for labels, answers, and pre-labeled data, helping reviewers understand how labelers are performing and how much of the work has been accepted, rejected, or remains unresolved.

General

• Effective time spent: The actual time labelers spent during the labeling process.

• Accepted pre-labeled: Pre-labeled labels or answers that were accepted by labelers.

• Rejected pre-labeled: Pre-labeled labels or answers that were not applied by labelers, whether it's rejected, ignored, or changed.

Labels

These metrics are available for span labeling, conversational labeling, and bounding box labeling projects.

• Accepted labels: Labels from labelers that were accepted by reviewers, either manually or through consensus.

• Rejected labels: Labels from labelers that were rejected by reviewers.

• Conflicted labels: Labels that have not yet been resolved through review or consensus.

• Missed labels: Labels added by reviewers in Reviewer mode that were not provided by labelers.

Answers

These metrics are available for row labeling and document labeling projects. If a project consists of multiple questions, each answer will be counted as one.

• Accepted answers: Answers from labelers that were accepted by reviewers, either manually or through consensus.

• Rejected answers: Answers from labelers that were rejected by reviewers.

• Answered rows: Number of rows that have been labeled. This metric is available for row labeling projects only.

For mixed labeling projects (projects with more than one labeling type), all relevant metrics are shown, including both labels and answers.

Distribution tab

The Distribution tab shows the distribution of labels and answers based on data from Labeler mode. Use it to identify class imbalance in your dataset and see which label classes or answer options are most frequently rejected during review.

Reading the panel

The panel shows one section for each labeling type in your project, such as Span labeling or Line labeling.

Within each section, results are grouped by label set (for example, Default NER) or question (for example, Yes or no?). Only the top three entries in each group are shown, ranked by count.

Each entry shows:

• Label class or answer: The label class or answer, with its total count shown in a badge.

• Accepted: The number of labels or answers accepted by reviewers.

• Rejected: The number of labels or answers rejected by reviewers.

Select View all to see every entry.

Full distribution

The Label and answer distribution dialog shows one tab for each labeling type in the project. Select a tab to view the complete distribution table for that labeling type.

For labeling types that use labels, span labeling, conversational labeling, and bounding box labeling:

Column	Description
Label set	The label set the class belongs to.
Label class	The label class, shown with its assigned color.
Count	The total number of times the class was applied.
Accepted	The number and percentage of labels of that class accepted by reviewers.
Rejected	The number and percentage of labels of that class rejected by reviewers.

For labeling types that use answers, row labeling, line labeling, and document labeling:

Column	Description
Question	The question the answer belongs to.
Answer	The selected answer option.
Count	The total number of times the answer was selected.
Accepted	The number and percentage of answers accepted by reviewers.
Rejected	The number and percentage of answers rejected by reviewers.

Every column can be sorted, and long tables are paginated.

For projects with a single labeling type, the dialog is titled according to its content. For example, Label distribution is used for a span labeling project and no tabs are shown.

Filtering and refreshing

• Use the document dropdown to switch between All documents and a specific document. This helps determine whether a distribution skew is dataset-wide or limited to certain files.

• Distribution data is cached. The panel and dialog show when the data was last updated.

  • Select Update now to recalculate the distribution data.