Analytics

Analytics to track labeler performance and progress in a project.

Analytics extension provides real-time performance metrics for each labeler in the project. You can use it to:

• View metrics and performance to track labeler performance and status, including accepted labels, rejected labels, etc.

• Filter by document to analyze performance in specific files.

• Filter by member to compare individual labelers and identify top performers or those who need support.

Reviewers can see performance data for all assigned labelers even before the project is marked as complete. This helps track progress throughout the labeling process.

Labelers can only see their own performance after the project is marked as complete by a reviewer.

Metrics

General

• Effective time spent: The actual time spent by labelers during the labeling process.

• Accepted pre-labeled: Pre-labeled labels/answers that are accepted by labelers.

• Rejected pre-labeled: Pre-labeled labels/answers that are not applied by labelers, whether it's rejected, ignored, or changed.

Labels

These metrics are available for span labeling, conversational labeling, and bounding box labeling projects.

• Accepted labels: Labels from labelers that are accepted by reviewers (manually or through consensus).

• Rejected labels: Labels from labelers that are rejected by reviewers.

• Conflicted labels: Labels that have not been resolved yet through review or consensus.

• Missed labels: Labels that were added by reviewers in reviewer mode but were not provided by the labelers.

Answers

These metrics are available for row labeling and document labeling projects. If the project consists of multiple questions, each answer will be counted as one.

• Accepted answers: Answers from labelers that are accepted by reviewers (manually or through consensus).

• Rejected answers: Answers from labelers that are rejected by reviewers.

• Answered rows: Number of rows that have been labeled (row labeling only).

For mixed labeling projects (projects with more than one labeling type), all relevant metrics are shown, including both labels and answers.