Common Terminology

A glossary of terms used throughout the app to help ensure a consistent understanding across all users.

Applied labels/answers

Labels or answers applied in the following contexts:

• Labeler mode: Includes manually applied labels, labels applied through assisted labeling features, and pre-labeled data.

• Reviewer mode: Includes labels or answers reviewed manually, applied through assisted labeling features, and pre-labeled data, but excludes labels automatically accepted through consensus.

Pre-labeled data: Labels or answers already included in the input file when the project is first created.

Assisted labeling features: Extensions that help apply labels automatically or in bulk, including Search (bulk label application to search results), ML-assisted labeling, Data programming, and Predictive labeling.

Counting method by labeling type

• Span labeling: Total number of labels applied to spans.

• Row or document labeling: Total number of answers submitted.

• Bounding box labeling: Total number of bounding boxes created.

• Conversational labeling: Total number of labels applied to spans, including those applied to each utterance.

• Mixed labeling: Combined total of all labels or answers calculated using the methods above.

Cell

A cell is a box that is used to display data in the text viewer. For example, in the image above, the box that contains the text Sherlock Holmes become widely popular in 1891 is a cell. Cells are arranged in a matrix-like structure.

• Cell line: The cell’s position on the vertical axis, numbered from 0 from top to bottom.

• Cell index: The cell’s position on the horizontal axis, numbered from 0 from left to right.

A cell is identified by its line and index. For example, the cell with line 3 and index 0 contains the text All but one are set in the Victorian or Edwardian eras, between about 1880 and 1914..

Characters and spans

Characters and spans are the basic units used in text annotation.

• Character: Typically the smallest unit in a document. This can be a letter, number, symbol, punctuation mark, or whitespace character.

• Span: A selected segment of text consisting of one or more characters, words, or phrases.

Spans are indexed starting from 0 within each cell, and characters within a span are also indexed starting from 0. For example, the span popularity has index 1 in the current cell and the character u has index 3 in popularity.

Consensus

Consensus is the required level of agreement among labelers before a label is automatically accepted. This helps ensure consistency and agreement across annotations.

For example, if the consensus threshold is set to 2:

1. Labeler 1 labels a span as PERSON.

2. The label is not automatically accepted yet because only one labeler has applied it.

3. Labeler 2 labels the same span as PERSON.

4. Since the required agreement level is reached, the label is automatically accepted.

Label set and label class

• Label set: A collection of related label classes. In the example above, Default NER is the name of the label set.

• Label class: A specific annotation label within a label set. In the example above, GEO, ORG, PER, and GPE are label classes.

  • Label clarification: A brief explanation of label classes and their intended use. In the example above, Names of organizations, such as companies, institutions, or agencies is a label class clarification.

Labeler

A labeler is responsible for labeling data across assigned projects. A labeler can be defined at two levels:

• Project level: Any workspace member can be assigned as a labeler in a project. See the labeler mode explanation below.

• Workspace level: A member assigned as a labeler in a workspace has limited workspace access and primarily works only on assigned projects.

Labeler mode and reviewer mode

Labeler mode and reviewer mode are isolated working environments within a project.

• Each labeler works independently in their own version of assigned documents.

• Reviewer mode contains a single shared version used to review, consolidate, and resolve conflicts across labelers’ work.

For example, in a project with 10 labelers and 3 reviewers:

• There are 10 separate labeler modes.

• There is 1 shared reviewer mode.

Project

A project is a configurable labeling workspace that contains documents and task settings used for annotation.

Reviewer

A reviewer is responsible for reviewing labelers’ work and resolving conflicts. Reviewer decisions become the final labeling result.

A reviewer can be defined at two levels:

• Project level: Any user assigned as a reviewer in a project.

• Workspace level: A user assigned as a reviewer in a workspace can access broader workspace functionality and resources. For a more detailed scope, see this page.