Datasaur
Visit our websitePricingBlogPlaygroundAPI Docs
  • Welcome to Datasaur
    • Getting started with Datasaur
  • Data Studio Projects
    • Labeling Task Types
      • Span Based
        • OCR Labeling
        • Audio Project
      • Row Based
      • Document Based
      • Bounding Box
      • Conversational
      • Mixed Labeling
      • Project Templates
        • Test Project
    • Creating a Project
      • Data Formats
      • Data Samples
      • Split Files
      • Consensus
      • Dynamic Review Capabilities
    • Pre-Labeled Project
    • Let's Get Labeling!
      • Span Based
        • Span + Line Labeling
      • Row & Document Based
      • Bounding Box Labeling
      • Conversational Labeling
      • Label Sets / Question Sets
        • Dynamic Question Set
      • Multiple Label Sets
    • Reviewing Projects
      • Review Sampling
    • Adding Documents to an Ongoing Project
    • Export Project
  • LLM Projects
    • LLM Labs Introduction
    • Sandbox
      • Direct Access LLMs
      • File Attachment
      • Conversational Prompt
    • Deployment
      • Deployment API
    • Knowledge base
      • External Object Storage
      • File Properties
    • Models
      • Amazon SageMaker JumpStart
      • Amazon Bedrock
      • Open AI
      • Azure OpenAI
      • Vertex AI
      • Custom model
      • Fine-tuning
      • LLM Comparison Table
    • Evaluation
      • Automated Evaluation
        • Multi-application evaluation
        • Custom metrics
      • Ranking (RLHF)
      • Rating
      • Performance Monitoring
    • Dataset
    • Pricing Plan
  • Workspace Management
    • Workspace
    • Role & Permission
    • Analytics
      • Inter-Annotator Agreement (IAA)
        • Cohen's Kappa Calculation
        • Krippendorff's Alpha Calculation
      • Custom Report Builder
      • Project Report
      • Evaluation Metrics
    • Activity
    • File Transformer
      • Import Transformer
      • Export Transformer
      • Upload File Transformer
      • Running File Transformer
    • Label Management
      • Label Set Management
      • Question Set Management
    • Project Management
      • Self-Assignment
        • Self-Unassign
      • Transfer Assignment Ownership
      • Reset Labeling Work
      • Mark Document as Complete
      • Project Status Workflow
        • Read-only Mode
      • Comment Feature
      • Archive Project
    • Automation
      • Action: Create Projects
  • Assisted Labeling
    • ML Assisted Labeling
      • Amazon Comprehend
      • Amazon SageMaker
      • Azure ML
      • CoreNLP NER
      • CoreNLP POS
      • Custom API
      • FewNERD
      • Google Vertex AI
      • Hugging Face
      • LLM Assisted Labeling
        • Prompt Examples
        • Custom Provider
      • LLM Labs (beta)
      • NLTK
      • Sentiment Analysis
      • spaCy
      • SparkNLP NER
      • SparkNLP POS
    • Data Programming
      • Example of Labeling Functions
      • Labeling Function Analysis
      • Inter-Annotator Agreement for Data Programming
    • Predictive Labeling
  • Assisted Review
    • Label Error Detection
  • Building Your Own Model
    • Datasaur Dinamic
      • Datasaur Dinamic with Hugging Face
      • Datasaur Dinamic with Amazon SageMaker Autopilot
  • Advanced
    • Script-Generated Question
    • Shortcuts
    • Extensions
      • Labels
      • Review
      • Document and Row Labeling
      • Bounding Box Labels
      • List of Files
      • Comments
      • Analytics
      • Dictionary
      • Search
      • Labeling Guidelines
      • Metadata
      • Grammar Checker
      • ML Assisted Labeling
      • Data Programming
      • Datasaur Dinamic
      • Predictive Labeling
      • Label Error Detection
      • LLM Sandbox
    • Tokenizers
  • Integrations
    • External Object Storage
      • AWS S3
        • With IRSA
      • Google Cloud Storage
      • Azure Blob Storage
      • Dropbox
    • SAML
      • Okta
      • Microsoft Entra ID
    • SCIM
      • Okta
      • Microsoft Entra ID
    • Webhook Notifications
      • Webhook Signature
      • Events
      • Custom Headers
    • Robosaur
      • Commands
        • Create Projects
        • Apply Project Tags
        • Export Projects
        • Generate Time Per Task Report
        • Split Document
      • Storage Options
  • API
    • Datasaur APIs
    • Credentials
    • Create Project
      • New mutation (createProject)
      • Python Script Example
    • Adding Documents
    • Labeling
      • Create Label Set
      • Add Label Sets into Existing Project
      • Get List of Label Sets in a Project
      • Add Label Set Item into Project's Label Set
      • Programmatic API Labeling
      • Inserting Span and Arrow Label into Document
    • Export Project
      • Custom Webhook
    • Get Data
      • Get List of Projects
      • Get Document Information
      • Get List of Tags
      • Get Cabinet
      • Export Team Overview
      • Check Job
    • Custom OCR
      • Importable Format
    • Custom ASR
    • Run ML-Assisted Labeling
  • Security and Compliance
    • Security and Compliance
      • 2FA
  • Compatibility & Updates
    • Common Terminology
    • Recommended Machine Specifications
    • Supported Formats
    • Supported Languages
    • Release Notes
      • Version 6
        • 6.111.0
        • 6.110.0
        • 6.109.0
        • 6.108.0
        • 6.107.0
        • 6.106.0
        • 6.105.0
        • 6.104.0
        • 6.103.0
        • 6.102.0
        • 6.101.0
        • 6.100.0
        • 6.99.0
        • 6.98.0
        • 6.97.0
        • 6.96.0
        • 6.95.0
        • 6.94.0
        • 6.93.0
        • 6.92.0
        • 6.91.0
        • 6.90.0
        • 6.89.0
        • 6.88.0
        • 6.87.0
        • 6.86.0
        • 6.85.0
        • 6.84.0
        • 6.83.0
        • 6.82.0
        • 6.81.0
        • 6.80.0
        • 6.79.0
        • 6.78.0
        • 6.77.0
        • 6.76.0
        • 6.75.0
        • 6.74.0
        • 6.73.0
        • 6.72.0
        • 6.71.0
        • 6.70.0
        • 6.69.0
        • 6.68.0
        • 6.67.0
        • 6.66.0
        • 6.65.0
        • 6.64.0
        • 6.63.0
        • 6.62.0
        • 6.61.0
        • 6.60.0
        • 6.59.0
        • 6.58.0
        • 6.57.0
        • 6.56.0
        • 6.55.0
        • 6.54.0
        • 6.53.0
        • 6.52.0
        • 6.51.0
        • 6.50.0
        • 6.49.0
        • 6.48.0
        • 6.47.0
        • 6.46.0
        • 6.45.0
        • 6.44.0
        • 6.43.0
        • 6.42.0
        • 6.41.0
        • 6.40.0
        • 6.39.0
        • 6.38.0
        • 6.37.0
        • 6.36.0
        • 6.35.0
        • 6.34.0
        • 6.33.0
        • 6.32.0
        • 6.31.0
        • 6.30.0
        • 6.29.0
        • 6.28.0
        • 6.27.0
        • 6.26.0
        • 6.25.0
        • 6.24.0
        • 6.23.0
        • 6.22.0
        • 6.21.0
        • 6.20.0
        • 6.19.0
        • 6.18.0
        • 6.17.0
        • 6.16.0
        • 6.15.0
        • 6.14.0
        • 6.13.0
        • 6.12.0
        • 6.11.0
        • 6.10.0
        • 6.9.0
        • 6.8.0
        • 6.7.0
        • 6.6.0
        • 6.5.0
        • 6.4.0
        • 6.3.0
        • 6.2.0
        • 6.1.0
        • 6.0.0
      • Version 5
        • 5.63.0
        • 5.62.0
        • 5.61.0
        • 5.60.0
  • Deployment
    • Self-Hosted
      • AWS Marketplace
        • Data Studio
        • LLM Labs
Powered by GitBook
On this page
  • Creating a Team
  • Invite Members
  • Creating a Project
  • Project Creation Step 4 - Assignment
  • Advanced setting
  • Project Creation Step 5 - Project Settings
  • Mask Personally Identifiable Information (PII)
  • Hide Labelers' Names
  • QA/Review
  • Team Overview
  • Project Analytics
  • Manage Tags in Project Team
  • Manage Export for Each Team Role
  • Remove team members
  1. Workspace Management

Workspace

Last updated 18 days ago

Project Creation Wizard Step Workforce management is a critical piece of the data labeling flow. We've interviewed 100+ ML teams around the world and captured best practices for managing a labeling team. Datasaur's workforce management platform allows you to create teams, invite team members, assign projects, and review the work.

Creating a Team

After signing in, you will automatically be directed to your personal workspace. On the top-right hand corner, please click Switch Workspace -> Create a New Workspace.. . You can type in a team name and add a logo.

Note: the maximum logo size that can be uploaded is 500 KB.

You can also rename the team and change the logo by hovering to the left sidebar and clicking Settings.

Invite Members

Once you have reached your team workspace, invite your team members by navigating to the Members page on the left sidebar and click "+ Invite member." In the pop-up, you can enter up to 20 email addresses (separated by commas) and choose a role for the new members, such as Admin, Supervisor, Reviewer, or Labeler.

Once done, click Send Invitation, and your teammates will receive an email with instructions to join.

If you ever need to update someone’s role, you can do that easily from the Members page. Just click the three-dot menu next to their name, pick the role you want to assign, and you’re all set.

You can also sort and filter members by name and role.

Creating a Project

Project Creation Step 4 - Assignment

The project creation step 4 is to properly set the assignment to labelers and reviewers.

  • A project can be assigned to one or more labelers.

  • Conflict Resolution:

    • The Peer Review Consensus slider allows you to determine the threshold at which labels will be automatically accepted. For highly sensitive projects where there is no room for error, you may want to ensure unanimity from all assigned labelers. For less sensitive projects where efficiency and cost are more important than accuracy, a majority vote may be sufficient. Any label where the threshold is not met will need to be manually reviewed by you, the project creator / reviewer.

    • If you check No Consensus, all of labelers label will be treated as conflicting labels

Advanced setting

If you have a project with multiple files and multiple labelers, using the Auto file distribution might save your time. Our system will distribute the files according to the consensus review that has been set previously.

💡 Best practice: a common practice is to assign multiple labelers to the same project to get multiple perspectives and eliminate human bias. This can increase accuracy, but will increase costs and time.

Project Creation Step 5 - Project Settings

In step 5, you can modify the project settings to match with your labeling workflow. These settings are also available in the Project Settings once the project is created.

  • Label set modification ensures that the label set cannot be changed by labelers.

  • Text modification ensures that the document text itself cannot be changed by labelers or reviewers. Kindly refer to the table below for the detailed permission.

    📌 Please note that only admin can access the setting.

  • Confirm unapplied label classes when marking document as complete ensures that the assignees can review all label classes that are not applied in the document when marking the document as complete.

  • Show labeler names in Review Mode ensures that the reviewers can see labelers' names.

  • Show rejected labels in Review Mode ensures that the reviewers can see all the labels they rejected.

  • Hide labels from inactive label set during review ensures that labels from inactive label set will be hidden during review.

  • Show original sentences in review mode ensures the reviewers can see all original sentences.

  • Mask Personally Identifiable Information (PII) is in the next section.

Mask Personally Identifiable Information (PII)

You can set the Mask PII setting to anonymize the selected information attributes in the project. After you turn the Mask PII setting on, you can define the masking data method and the attributes to mask by clicking the Data Masking Settings.

Masking Data Method is how Datasaur masks the information attributes that you have defined. There are two ways to mask the information attributes:

  • Random Character: Replace the personal information with random character.

    • Example: May 23rd, 2022 → Pgh 78ax, 9928

  • Asterisk: Replace the personal information with asterisk.

    • Example: John Doe -> **** ***

To select the information attributes to mask, you can check/uncheck the PII fields on the table. You can also set the regular expression rule to define what entities to mask.

Hide Labelers' Names

As an admin, you can also hide labelers' names from the reviewers. This feature can be useful to avoid bias while reviewing the project. You can disable the Show labeler names in Review Mode toggle to turn this feature on. The Review extension will look like the screenshot below to the Reviewer.

QA/Review

After clicking the Launch Project button, you will be taken to the newly created project.

âš  Note that as the project creator you will start in Reviewer Mode. If you would like to start labeling, you will need to assign yourself, click the profile dropdown on the navigation bar, then click Switch to Labeler Mode. Any other assigned labelers will automatically start in Labeler Mode.

Reviewer Mode has a Review extension on the right side. This will contain a Conflicts section.

Team Overview

Team overview contains a high-level overview of your team's projects and progress. This dashboard is only available and visible to Team Admins.

Project Analytics

Manage Tags in Project Team

Note: only admins can manage tags.

Manage Export for Each Team Role

This feature allows you to set the limitations for each role in exporting the project.

For example, you want to disable Export feature on labelers' accounts due to security reasons. You can simply uncheck all the methods on Labeler section, then click Save.

After that, Export feature will not be shown on labeler's account like the example below.

Remove team members

Removing team members can be done by navigating to the left sidebar and select Members page. You can click the triple dots and click Remove.

Don't worry! Removing team members from the workspace will remain all their labeling work.

Creating a project in a team workspace can only be done by an admin. It is the same as creating a project in your personal workspace, with one key difference. When creating a custom project as an admin, there are additional 4th and 5th steps. Please follow the steps outlined in until you reach step 4.

Allow dynamic review assignment allows you assign your team member as a reviewer automatically when the labelers have conflicts in a project. The detailed information can be found .

Advanced setting allows you to distribute files to each of the labelers and into multiple parts.

For more information about Document Labeling Reviewer Mode, please refer to this .

Datasaur has 10 different analytic measurement in this page. The detailed information about each analytic can be seen .

Beside the analytics for all projects in page, Datasaur also provide the detailed analytics per project. You can find the page by click the triple dots on each project, then choose View Project Analytics.

In addition to being able to , you can also manage tags for a project team from the Team Settings page. You can navigate to Settings on the left-side bar and click on Tags.

Creating a Custom Project
here
split the files
link
here
Overview
manage tags from the Projects Dashboard
Datasaur Project List Page
Datasaur Setting Page
Invite Member
Project Creation Wizard Step 4
Advanced setting in Step 4
Project Creation Wizard Step 5
Text Modification
Team Overview