Krippendorff's Alpha Calculation

Explain how Datasaur implements the Krippendorff's Alpha algorithm.

Krippendorff's Alpha is one of the algorithms that is supported by Datasaur to calculate the agreement while taking into account the possibility of chance agreement. We will deep dive into how Datasaur collects all labels from labelers and reviewers in a project and process them into an Inter-annotator Agreement matrix.

Sample Data

Suppose there are 2 labelers and 1 reviewer — Labeler A, Labeler B, and Reviewer — who labeled the same spans. Labeler A work is visualized in Image 1, Labeler B work is visualized in Image 2, and Reviewer work is visualized in Image 3.

Image 1. Labeler A Work

Image 2. Labeler B Work

Image 3. Reviewer Work

Calculating the Agreement

In this section, we will see the calculation detail between Labeler A and Reviewer.

1. Arranging the data

First, we need to arrange the sample data into Table 1 for the better visualization.

Table 1. Sample Data

Span	Labeler A	Reviewer
The Tragedy of Hamlet	EVE	TITLE
Prince of Denmark	PER
Hamlet	PER	PER
William Shakespeare	PER	PER
1599	YEAR	YEAR
1601	YEAR	YEAR
Shakespeare	ORG	PER
30557		QTY

2. Cleaning the data

Second, we need to remove spans that only have 1 label i.e. Prince of Denmark and 30557. They should be removed because spans with a single label will introduce a calculation error. The calculation result will still show the agreement level between 2 annotators. The cleaned data is shown in Table 2.

Table 2. Cleaned Data

Span	Labeler A	Reviewer
The Tragedy of Hamlet	EVE	TITLE
Hamlet	PER	PER
William Shakespeare	PER	PER
1599	YEAR	YEAR
1601	YEAR	YEAR
Shakespeare	ORG	PER

3. Creating the agreement table

Third, we need to create an agreement table based on the cleaned data. The table is visualized in Table 3.

Table 3. Agreement Table

Based on the table, 5 values are calculated: $n$, $r_i$, $r_k$, $r$, and $r'$.

Total spans in the data

• $n$ is the total spans in the data.

  • Here, $n=6$ because there are 6 spans.

Total labels in each span

$$r_i=\sum\limits_{k=1}^{m}r_{ik} (1)$$

• $r_i$ is the total labels that span $i$ has.

• $m$ is the total number of label.

  • Here, $m=5$ because there are 5 labels.

• $r_{ik}$ is the number of $k$ label in span $i$.

Here is the calculation result.

• $r_1=r_{1,EVE}+r_{1,ORG}+r_{1,PER}+r_{1,TITLE}+r_{1,YEAR}=1+0+0+1+0=2$

• $r_2=r_{2,EVE}+r_{2,ORG}+r_{2,PER}+r_{2,TITLE}+r_{2,YEAR}=0+0+2+0+0=2$

• $r_3=r_{3,EVE}+r_{3,ORG}+r_{3,PER}+r_{3,TITLE}+r_{3,YEAR}=0+0+2+0+0=2$

• $r_4=r_{4,EVE}+r_{4,ORG}+r_{4,PER}+r_{4,TITLE}+r_{4,YEAR}=0+0+0+0+2=2$

• $r_5=r_{5,EVE}+r_{5,ORG}+r_{5,PER}+r_{5,TITLE}+r_{5,YEAR}=0+0+0+0+2=2$

• $r_6=r_{6,EVE}+r_{6,ORG}+r_{6,PER}+r_{6,TITLE}+r_{6,YEAR}=0+1+1+0+0=2$

Total of each label

$$r_k=\sum\limits_{i=1}^{n}r_{ik} (2)$$

• $r_k$ is the total of $k$ label in the data.

• $n$ is the total spans in the data.

• $r_{ik}$ is the number of $k$ label in span $i$.

Here is the calculation result.

• $r_{EVE}=r_{1,EVE}+r_{2,EVE}+r_{3,EVE}+r_{4,EVE}+r_{5,EVE}+r_{6,EVE}=1+0+0+0+0+0=1$

• $r_{ORG}=r_{1,ORG}+r_{2,ORG}+r_{3,ORG}+r_{4,ORG}+r_{5,ORG}+r_{6,ORG}=0+0+0+0+0+1=1$

• $r_{PER}=r_{1,PER}+r_{2,PER}+r_{3,PER}+r_{4,PER}+r_{5,PER}+r_{6,PER}=0+2+2+0+0+1=5$

• $r_{TITLE}=r_{1,TITLE}+r_{2,TITLE}+r_{3,TITLE}+r_{4,TITLE}+r_{5,TITLE}+r_{6,TITLE}=1+0+0+0+0+0=1$

• $r_{YEAR}=r_{1,YEAR}+r_{2,YEAR}+r_{3,YEAR}+r_{4,YEAR}+r_{5,YEAR}+r_{6,YEAR}=0+0+0+2+2+0=4$

Total labels in the data

$$r=\sum\limits_{i=1}^nr_i (3)$$

• $r$ is the total labels in the data.

• $n$ is the total spans in the data.

• $r_i$ is the total labels that span $i$ has.

Here is the calculation result.

• $r=r_1+r_2+r_3+r_4+r_5+r_6=12$

Average number of labels per span

$$r'=\frac{r}{n} (4)$$

• $r'$ is the average number of labels per span.

• $n$ is the total spans in the data.

Here is the calculation result.

• $r'=\frac{r}{n}=\frac{12}{6}=2$

4. Choosing weight function

Fourth, we need a weight function to weight the labels. Every label is treated equally because one label is no difference than the other. Hence, the weight function that will be used is stated in Formula 5.

$$w_{ik}=r_{ik} (5)$$

• $w_{ik}$ is the weighted number of $k$ label in span $i$.

• $r_{ik}$ is the number of $k$ label in span $i$.

5. Calculating Pa

Fifth, the observed weighted percent agreement is calculated.

Weighted number of labels

We will start by calculating the weighted number of label using Formula (6).

$$r_{ik+}=\sum\limits_{l=1}^{m} w_{kl}r_{il} (6)$$

• $r_{ik+}$ is the weighted number of $k$ label in span $i$.

• $m$ is the total number of label.

• $w_{kl}$ is the weighted number of $l$ label in span $k$.

• $r_{il}$ is the number of $l$ label in span $i$.

For example, we can apply Formula (6) to calculate the weighted EVE label in span 1.

$$r_{1,EVE+}=\sum\limits_{l=1}^{5} w_{EVE,l}r_{1,l}=1*1+0*0+0*0+0*1+0*0=1$$

We need to calculate all of the span and label combination. The complete calculation result is visualized in Table 4.

Table 4. Weighted Number of Label

Agreement percentage

After we got the weighted number of labels, we need to calculate the agreement percentage for a single span and label using Formula (7).

$$p_{a|ik}=\frac{r_{ik}(r_{ik+}-1)}{r'(r_i-1)} (7)$$

• $p_{a|ik}$ is the agreement percentage of $k$ label in span $i$.

• $r_{ik}$ is the number of $k$ label in span $i$.

• $r_{ik+}$ is the weighted number of $k$ label in span $i$.

• $r'$ is the average number of labels per span.

• $r_i$ is the total labels that span $i$ has.

For example, we can apply Formula (7) to calculate the agreement percentage of EVE label in span 1.

$$p_{a|1,EVE}=\frac{r_{1,EVE}(r_{1,EVE+}-1)}{r'(r_1-1)}=\frac{1(1-1)}{2(2-1)}=0$$

We need to calculate all of the span and label combination. The complete calculation result is visualized in Table 5.

Table 5. Agreement Percentage

Agreement percentage of a single span

We can simplify the result by getting the agreement percentage of a single span using Formula (8).

$$p_{a|i}=\sum\limits_{k=1}^{m} p_{a|ik} (8)$$

• $p_{a|i}$ is the agreement percentage of span $i$.

• $m$ is the total number of label.

• $p_{a|ik}$ is the agreement percentage of $k$ label in span $i$.

For example, we can apply Formula (8) to calculate the agreement percentage of span 1.

$$p_{a|1}=\sum\limits_{k=1}^{5} p_{a|1,k}=0+0+0+0+0=0$$

We need to calculate the agreement percentage of all spans. The complete calculation result is visualized in Table 6.

Table 6. Agreement Percentage of Each Span

Average agreement percentage

From the previous calculation, we can calculate the average agreement percentage using Formula (9).

$$p_a'=\frac{1}{n}\sum\limits_{i=1}^{n}P_{a|i} (9)$$

• $p_a'$ is the average agreement percentage.

• $n$ is the total spans in the data.

• $p_{a|i}$ is the agreement percentage of span $i$.

We can apply Formula (9) to calculate the average agreement percentage.

$$p_a'=\frac{1}{6}\sum\limits_{i=1}^{6}P_{a|i}=\frac{1}{6}(0+1+1+1+1+0)=0.6666$$

Calculating Pa

Finally, the observed weighted percent agreement is calculated using Formula (10).

$$p_a=p_a'(1-\frac{1}{nr'})+\frac{1}{nr'} (10)$$

• $p_a$ is the observed weighted percent agreement.

• $p_a'$ is the average agreement percentage.

• $n$ is the total spans in the data.

• $r'$ is the average number of labels per span.

We can apply Formula (10) to calculate the observed weighted agreement percentage.

$$p_a=p_a'(1-\frac{1}{nr'})+\frac{1}{nr'}=0.6666(1-\frac{1}{6\times2})+\frac{1}{6\times2}=0.6944$$

6. Calculating Pe

Sixth, the chance weighted percent agreement is calculated.

Classification probability

We start by calculating the classification probability for each label using Formula (11).

$$\pi_k=\frac{r_k}{r} (11)$$

• $\pi_k$ is the classification probability for $k$ label.

• $r_k$ is the total of $k$ label in the data.

• $r$ is the total labels in the data.

Here is the calculation result.

• $\pi_{EVE}=\frac{r_{EVE}}{r}=\frac{1}{12}=0.0833$

• $\pi_{ORG}=\frac{r_{ORG}}{r}=\frac{1}{12}=0.0833$

• $\pi_{PER}=\frac{r_{PER}}{r}=\frac{5}{12}=0.4166$

• $\pi_{TITLE}=\frac{r_{TITLE}}{r}=\frac{1}{12}=0.0833$

• $\pi_{YEAR}=\frac{r_{YEAR}}{r}=\frac{4}{12}=0.3333$

Calculating Pe

To calculate the chance weighted percent agreement, Formula (11) can be applied to Formula (12).

$$p_e=\sum\limits_{k=1}^{m}{\pi_k}^2 (12)$$

• $p_e$ is the chance weighted percent agreement.

• $m$ is the total number of label.

• $\pi_k$ is the classification probability for $k$ label.

Here is the chance weighted percent agreement calculation.

$p_e=\sum\limits_{k=1}^{m}{\pi_k}^2$

$p_e={\pi_{EVE}}^2+{\pi_{ORG}}^2+{\pi_{PER}}^2+{\pi_{TITLE}}^2+{\pi_{YEAR}}^2$

$p_e=0.0833^2+0.0833^2+0.4166^2+0.0833^2+0.3333^2$

$p_e=0.3055$

7. Calculating the Alpha

Finally, Krippendorff's alpha is calculated using Formula (13).

$$\alpha=\frac{p_a-p_e}{1-p_e} (13)$$

• $\alpha$ is the Krippendorff's alpha between Labeler A and Reviewer.

• $p_a$ is the observed weighted percent agreement.

• $p_e$ is the chance weighted percent agreement.

We can get the $\alpha$ by applying $p_a$ and $p_e$ to Formula (13).

$$\alpha=\frac{p_a-p_e}{1-p_e}=\frac{0.6944-0.3055}{1-0.3055}=0.56$$

Summary

• We apply the same calculation for agreement between labelers, and between reviewer and labelers.

• Missing labels from a single labeler will be removed.

• The percentage of chance agreement will vary depending on:

  • The number of the labels in a project.

  • The number of label options.

• When both labelers agree but the reviewer rejects the labels:

  • The agreement between the two labelers increases.

  • The agreement between the labelers and the reviewer decreases.