# Cohen's Kappa Calculation
[Cohen's Kappa](https://en.wikipedia.org/wiki/Cohen's_kappa) is one of the algorithms that is supported by Datasaur to calculate the agreement while taking into account the possibility of chance agreement. We will dive deep into how Datasaur collects all labels from labelers and reviewers in a project and processes them into an Inter-annotator Agreement matrix.
## Sample Data
Suppose there are 2 labelers—Labeler A and Labeler B—who labeled the same sentences.
There is also a reviewer who labeled the same sentences.
## Calculating the Data
### Agreement Records
Based on the screenshots above, we map those labels into the agreement records below:
| Position in sentence | Labeler A | Labeler B | Reviewer |
| --------------------- | --------- | --------- | -------- |
| The Tragedy of Hamlet | EVE | TITLE | TITLE |
| Prince of Denmark | PER | \ | \ |
| Hamlet | PER | TITLE | PER |
| William Shakespeare | PER | PER | PER |
| 1599 | YEAR | YEAR | YEAR |
| 1601 | YEAR | YEAR | YEAR |
| Shakespeare | ORG | ORG | PER |
| 30,557 | \ | \ | QTY |
### **Agreement Table / Confusion Matrix**
Then, we construct the records into the agreement table. We use the data from Labeler A and Labeler B for the simulation.
### Calculating the Kappa
From the table above, there are **7** records with **4** agreements.
.png?alt=media)
The observed proportionate agreement is:
.png?alt=media)
To calculate the probability of random agreement, we note that:
* Labeler A labeled `EVE` once and Labeler B didn't label `EVE`. Therefore, the probability of random agreement on the label `EVE` is:
%20\(1\)%20\(1\)%20\(1\).png?alt=media)
* Compute the probability of random agreement for all labels:
.png?alt=media)
The full random agreement probability is the sum of the probability of random agreement for all labels:
.png?alt=media)
Finally, we can calculate the Cohen's Kappa:
.png?alt=media)
#### **Kappa for Labeler A and Reviewer**
#### **Kappa for Labeler B and Reviewer**
## Summary
* We apply the same calculation for agreement between labelers and between the reviewer and labelers.
* Missing labels from a single labeler will be counted as having applied empty labels.
* The percentage of chance agreement will vary depending on:
* The number of the labels in a project.
* The number of label options.
* When both labelers agree but the reviewer rejects the labels:
* The agreement between the two labelers increases.
* The agreement between the labelers and the reviewer decreases.
---
# Agent Instructions: Querying This Documentation
If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question.
Perform an HTTP GET request on the current page URL with the `ask` query parameter:
```
GET https://docs.datasaur.ai/workspace-management/analytics/inter-annotator-agreement/cohens-kappa-calculation.md?ask=
```
The question should be specific, self-contained, and written in natural language.
The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.
Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.
.png?alt=media)