> For the complete documentation index, see [llms.txt](https://docs.datasaur.ai/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.datasaur.ai/workspace-management/analytics/inter-annotator-agreement.md).

# Inter-Annotator Agreement (IAA)

Inter-annotator agreement (IAA) measures how consistently multiple annotators make the same labeling decisions for a given label category or class. It helps evaluate the clarity of your guidelines and the reproducibility of your results.

We support two algorithms to calculate the agreement between two annotators:

* [Cohen's Kappa](/workspace-management/analytics/inter-annotator-agreement/cohens-kappa-calculation.md)
* [Krippendorff's Alpha](/workspace-management/analytics/inter-annotator-agreement/krippendorffs-alpha-calculation.md)

Note that we apply the scale interpretation of Krippendorff's Alpha for both methods as in Image 1.

![Krippendorff's alpha scale](https://docs.google.com/drawings/u/0/d/skeGno0Afrw0ZIKAZpYbZ2A/image?w=513\&h=93\&rev=137\&ac=1\&parent=1UmG4KLgmliNMX-9QkW8wSX64xkFfe-GlU7jp5IM8lDI)

* **Discard** will be presented in red.
* **Tentative** will be presented in yellow.
* **Good** will be presented in green.

IAA is calculated in the background as soon as a project status changed to **Ready for review** (after all labelers mark the project as complete) or **Complete** (a reviewer marks the project as complete).

## View IAA data

You can view inter-annotator agreement at both the workspace and project levels to analyze consistency across your labeling work. To view IAA across projects:

1. Go to **Analytics > Team Overview** from the left sidebar.
2. Open the **Inter-Annotator Agreement** tab.

Only **admin** can view this data.

### Workspace level

At the workspace level, IAA is calculated as a weighted average across all projects. It does not recalculate all labels or answers. Instead, it considers how often each label or answer occurs between annotators, providing a more accurate representation of overall agreement.

<figure><img src="https://448889121-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-MbjY0HseEqu7LtYAt4d%2Fuploads%2Fgit-blob-80840320523a3112b48483bf18e73599af2b0672%2FAnalytics%20-%20IAA%20-%20Cohen's%20Kappa%20-%20All%20projects.png?alt=media" alt=""><figcaption><p>Team overview IAA</p></figcaption></figure>

### Project level

To view IAA at the project level, select a project from the **Project** field.

<figure><img src="https://448889121-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-MbjY0HseEqu7LtYAt4d%2Fuploads%2Fgit-blob-40df0fc741aad5c06206b822a5bdc8bf79dc65e1%2FAnalytics%20-%20IAA%20-%20Cohen's%20Kappa%20-%20specific%20project.png?alt=media" alt=""><figcaption></figcaption></figure>

You can also view it from the **Project analytics** page:

1. Go to the **Projects** page.
2. Click the **three-dot menu** on a project and select **View project analytics**.
3. Open the **Inter-Annotator Agreement** tab.