Datasaur
Visit our websitePricingBlogPlaygroundAPI Docs
  • Welcome to Datasaur
    • Getting started with Datasaur
  • Data Studio Projects
    • Labeling Task Types
      • Span Based
        • OCR Labeling
        • Audio Project
      • Row Based
      • Document Based
      • Bounding Box
      • Conversational
      • Mixed Labeling
      • Project Templates
        • Test Project
    • Creating a Project
      • Data Formats
      • Data Samples
      • Split Files
      • Consensus
      • Dynamic Review Capabilities
    • Pre-Labeled Project
    • Let's Get Labeling!
      • Span Based
        • Span + Line Labeling
      • Row & Document Based
      • Bounding Box Labeling
      • Conversational Labeling
      • Label Sets / Question Sets
        • Dynamic Question Set
      • Multiple Label Sets
    • Reviewing Projects
      • Review Sampling
    • Adding Documents to an Ongoing Project
    • Export Project
  • LLM Projects
    • LLM Labs Introduction
    • Sandbox
      • Direct Access LLMs
      • File Attachment
      • Conversational Prompt
    • Deployment
      • Deployment API
    • Knowledge base
      • External Object Storage
      • File Properties
    • Models
      • Amazon SageMaker JumpStart
      • Amazon Bedrock
      • Open AI
      • Azure OpenAI
      • Vertex AI
      • Custom model
      • Fine-tuning
      • LLM Comparison Table
    • Evaluation
      • Automated Evaluation
        • Multi-application evaluation
        • Custom metrics
      • Ranking (RLHF)
      • Rating
      • Performance Monitoring
    • Dataset
    • Pricing Plan
  • Workspace Management
    • Workspace
    • Role & Permission
    • Analytics
      • Inter-Annotator Agreement (IAA)
        • Cohen's Kappa Calculation
        • Krippendorff's Alpha Calculation
      • Custom Report Builder
      • Project Report
      • Evaluation Metrics
    • Activity
    • File Transformer
      • Import Transformer
      • Export Transformer
      • Upload File Transformer
      • Running File Transformer
    • Label Management
      • Label Set Management
      • Question Set Management
    • Project Management
      • Self-Assignment
        • Self-Unassign
      • Transfer Assignment Ownership
      • Reset Labeling Work
      • Mark Document as Complete
      • Project Status Workflow
        • Read-only Mode
      • Comment Feature
      • Archive Project
    • Automation
      • Action: Create Projects
  • Assisted Labeling
    • ML Assisted Labeling
      • Amazon Comprehend
      • Amazon SageMaker
      • Azure ML
      • CoreNLP NER
      • CoreNLP POS
      • Custom API
      • FewNERD
      • Google Vertex AI
      • Hugging Face
      • LLM Assisted Labeling
        • Prompt Examples
        • Custom Provider
      • LLM Labs (beta)
      • NLTK
      • Sentiment Analysis
      • spaCy
      • SparkNLP NER
      • SparkNLP POS
    • Data Programming
      • Example of Labeling Functions
      • Labeling Function Analysis
      • Inter-Annotator Agreement for Data Programming
    • Predictive Labeling
  • Assisted Review
    • Label Error Detection
  • Building Your Own Model
    • Datasaur Dinamic
      • Datasaur Dinamic with Hugging Face
      • Datasaur Dinamic with Amazon SageMaker Autopilot
  • Advanced
    • Script-Generated Question
    • Shortcuts
    • Extensions
      • Labels
      • Review
      • Document and Row Labeling
      • Bounding Box Labels
      • List of Files
      • Comments
      • Analytics
      • Dictionary
      • Search
      • Labeling Guidelines
      • Metadata
      • Grammar Checker
      • ML Assisted Labeling
      • Data Programming
      • Datasaur Dinamic
      • Predictive Labeling
      • Label Error Detection
      • LLM Sandbox
    • Tokenizers
  • Integrations
    • External Object Storage
      • AWS S3
        • With IRSA
      • Google Cloud Storage
      • Azure Blob Storage
    • SAML
      • Okta
      • Microsoft Entra ID
    • SCIM
      • Okta
      • Microsoft Entra ID
    • Webhook Notifications
      • Webhook Signature
      • Events
      • Custom Headers
    • Robosaur
      • Commands
        • Create Projects
        • Apply Project Tags
        • Export Projects
        • Generate Time Per Task Report
        • Split Document
      • Storage Options
  • API
    • Datasaur APIs
    • Credentials
    • Create Project
      • New mutation (createProject)
      • Python Script Example
    • Adding Documents
    • Labeling
      • Create Label Set
      • Add Label Sets into Existing Project
      • Get List of Label Sets in a Project
      • Add Label Set Item into Project's Label Set
      • Programmatic API Labeling
      • Inserting Span and Arrow Label into Document
    • Export Project
      • Custom Webhook
    • Get Data
      • Get List of Projects
      • Get Document Information
      • Get List of Tags
      • Get Cabinet
      • Export Team Overview
      • Check Job
    • Custom OCR
      • Importable Format
    • Custom ASR
    • Run ML-Assisted Labeling
  • Security and Compliance
    • Security and Compliance
      • 2FA
  • Compatibility & Updates
    • Common Terminology
    • Recommended Machine Specifications
    • Supported Formats
    • Supported Languages
    • Release Notes
      • Version 6
        • 6.111.0
        • 6.110.0
        • 6.109.0
        • 6.108.0
        • 6.107.0
        • 6.106.0
        • 6.105.0
        • 6.104.0
        • 6.103.0
        • 6.102.0
        • 6.101.0
        • 6.100.0
        • 6.99.0
        • 6.98.0
        • 6.97.0
        • 6.96.0
        • 6.95.0
        • 6.94.0
        • 6.93.0
        • 6.92.0
        • 6.91.0
        • 6.90.0
        • 6.89.0
        • 6.88.0
        • 6.87.0
        • 6.86.0
        • 6.85.0
        • 6.84.0
        • 6.83.0
        • 6.82.0
        • 6.81.0
        • 6.80.0
        • 6.79.0
        • 6.78.0
        • 6.77.0
        • 6.76.0
        • 6.75.0
        • 6.74.0
        • 6.73.0
        • 6.72.0
        • 6.71.0
        • 6.70.0
        • 6.69.0
        • 6.68.0
        • 6.67.0
        • 6.66.0
        • 6.65.0
        • 6.64.0
        • 6.63.0
        • 6.62.0
        • 6.61.0
        • 6.60.0
        • 6.59.0
        • 6.58.0
        • 6.57.0
        • 6.56.0
        • 6.55.0
        • 6.54.0
        • 6.53.0
        • 6.52.0
        • 6.51.0
        • 6.50.0
        • 6.49.0
        • 6.48.0
        • 6.47.0
        • 6.46.0
        • 6.45.0
        • 6.44.0
        • 6.43.0
        • 6.42.0
        • 6.41.0
        • 6.40.0
        • 6.39.0
        • 6.38.0
        • 6.37.0
        • 6.36.0
        • 6.35.0
        • 6.34.0
        • 6.33.0
        • 6.32.0
        • 6.31.0
        • 6.30.0
        • 6.29.0
        • 6.28.0
        • 6.27.0
        • 6.26.0
        • 6.25.0
        • 6.24.0
        • 6.23.0
        • 6.22.0
        • 6.21.0
        • 6.20.0
        • 6.19.0
        • 6.18.0
        • 6.17.0
        • 6.16.0
        • 6.15.0
        • 6.14.0
        • 6.13.0
        • 6.12.0
        • 6.11.0
        • 6.10.0
        • 6.9.0
        • 6.8.0
        • 6.7.0
        • 6.6.0
        • 6.5.0
        • 6.4.0
        • 6.3.0
        • 6.2.0
        • 6.1.0
        • 6.0.0
      • Version 5
        • 5.63.0
        • 5.62.0
        • 5.61.0
        • 5.60.0
  • Deployment
    • Self-Hosted
      • AWS Marketplace
        • Data Studio
        • LLM Labs
Powered by GitBook
On this page
  • Custom Report Builder
  • Analytics Extension on a Project
  • Charts
  • Multiple Ways to View the Data
  • Overview
  • Project
  • Team Member
  • Evaluation Metrics
  1. Workspace Management

Analytics

Last updated 3 months ago

Datasaur offers various methods to efficiently view and analyze your data. The Analytics pages are exclusively accessible to Administrators. For convenient data access, you can choose to export it, and it will be delivered to your email.

Custom Report Builder

We also have a feature to customize your own report to get the Analytics data that you want. For more detailed information, you can access .

Analytics Extension on a Project

Other pages offer comprehensive performance information, focusing on the results of labeling work. To cover all the bases, Datasaur will make sure it's easy to track each labeler's progress during an ongoing project. You can easily see how many labels have been produced, how many questions have been answered, and how many documents are left for each labeler.

For more details, visit .

Charts

Here are some tips to make sense and interpret the following charts below.

  1. Higher values on the charts indicate better performance. This means that your team is consistently improving their speed, accuracy, and overall efficiency.

  2. It's important to note that the total labels shown on the Throughput chart may not always match the total labels on the Quality chart. A higher value on the Throughput chart indicates that there are labels manually applied in Reviewer Mode, as the Quality chart only calculates labels from Labeler Mode. This could also indicate potential issues if there is a significant difference, as it may suggest that the Reviewer had to manually label a large amount of data.

  3. In some cases, you may observe high Efficiency values while both Throughput and Quality values are low. This typically occurs when doing projects with lots of pre-labeled data, as Throughput and Quality calculations only consider manually applied labels and exclude pre-labeled data.

Overall Projects

Display the current total projects distribution based on its status. Note that is a statistic and not a time-series data.

Remaining Files

Display the current remaining files from uncompleted projects, break down by project status. Note that is this one is also statistic and not a time-series data.

Throughput

It demonstrates the speed at which your team can produce annotations (labels and answers). It is calculated by summing the following factors:

  1. Total labels applied from each of labelers.

  2. Total labels applied from the Reviewer Mode. However, this count excludes labels that are automatically accepted through consensus. It focuses solely on labels applied manually by reviewers, including those involved in conflict resolution.

Efficiency

It illustrates the effectiveness of your labeling process in generating accepted labels per minute.

It is calculated by dividing the total number of accepted labels from Reviewer Mode (including those resolved manually, applied directly, through consensus, and pre-labeled data) by the total time spent by all team members, broken down on a daily basis.

Quality

  1. Total accepted labels This metric represents the sum of all labels applied from each labelers that have been accepted through both consensus and manual review processes.

  2. Total rejected labels This metric represents the sum of all labels applied from each labelers that have been rejected through both consensus and manual review processes.

  3. Total unresolved conflicts This metric represents the sum of all labels from each labelers that have not yet been accepted (either manually or through consensus) or rejected.

Cumulative Time Spent

It gives the total time spent of each day, categorized by labeler and reviewer role. This metric is computed when team members have a project open on their active browser tab. If they switch to another tab while working on the project, that time will not be factored into the calculation.

A project is considered idle if there is no mouse, keyboard, or scroll movement for two minutes. When a project is open and active, it will count the time spent by sending a request to the backend every 60 seconds. If a workspace becomes idle before 60 seconds have passed, it will send the elapsed time since the last request to the backend.

Multiple Ways to View the Data

Overview

You can use the project tag filter to view detailed analytics for specific projects. With this functionality, you have the option to select multiple tags, enabling the filters to operate using OR logic. This means that any projects containing at least one of the selected tags will be displayed in the filter results.

Project

Quickly access essential project information through the Name column, including statistics like total files, total time spent, token count (Span Labeling), and row count (Row Labeling). Gain additional insights by hovering over the avatar icon in the Labelers column or the Status column (only for in review and labeling in progress) to get the snapshot of the labeling progress.

Team Member

Explore specific team member's data and insights, providing a comprehensive view of their contributions and performance based on the role. Once again, we will use the same charts above but filter specifically for a particular team member. You can also see the overview information for each project assigned.

Regarding the table, it displays the latest statistics and is not dependent on the date range selected. It's important to note that the data in the table is specific to either the Labeler or Reviewer Mode, which can be toggled using the tab at the top of the page.

Access this report by going to your Members page, clicking on the three dots corresponding to your teammate, and then choosing View Member Details.

Evaluation Metrics

It provides a breakdown of conflicts that occurred among labelers and how they were handled by the reviewers. This metric could also give you insights regarding the level of agreement among labelers (especially if combined with ) and the effectiveness of the reviewers in resolving conflicts. It's important to note that conflicts do not occur for Bounding Box Labeling and LLM projects. The data is categorized as follows:

Gain a high-level understanding of your data within the Workspace. This page calculates metrics from all your projects and provides overall comprehensive insights, including the . Access the Overview page by selecting Analytics on the sidebar.

For an in-depth analysis of a specific project, click the triple dot next to the project and select the "View project analytics". This detailed page includes a summary of your labeling work in the , specific charts for the project, across all assignees, and comparing Reviewer Mode with each Labeler Mode.

To assist you in evaluating the labeling, Datasaur calculates the evaluation metrics, which are available specifically for each completed project. For more detailed information, you can access .

IAA
Inter-Annotator Agreement
report tab
IAA
evaluation metrics
this page
this page
this page