# Inter-Annotator Agreement for Data Programming
**Data programming** empowers you to calculate the performance of your models and final answers using inter-annotator agreement (IAA) calculation.
To learn more about inter-annotator agreement, please visit [the following link](https://datasaurai.gitbook.io/datasaur/basics/workforce-management/analytics/inter-annotator-agreement).
Below are the steps to evaluate the performance.
1. Create a project
2. Activate **Data programming** through the **Manage extensions** dialog.
Activate Data Programming
3. Create labeling functions for the selected question. You need a minimum of two labeling functions to obtain the IAA value.
Data Programming Editor
**Special notes**: If you use prelabeled columns as the representative of your models, you can create the labeling function like below :
* If using Snorkel provider, please use this code
```python
@labeling_function()
def labeling_function(x) -> int:
# Implement your logic here
text = x.columns[x.column_name_to_index['column_name']]
for key, value in LABELS.items():
if re.search(key, text, re.IGNORECASE):
return value
return ABSTAIN
```
* If using Stegosaurus provider, please use this code and activate **Multiple-label template**:
```python
@target_label()
ABSTAIN = -1
def label_function(sample):
text = sample['column_name']
# text = sample[COLUMN_NAME] if only want to use content from certain column
# Implement your logic here
# Keywords value on the certain column
DICT_KEYWORDS = {
'positive' : ['positive'],
'negative' : ['negative']
}
for label, target_keywords in DICT_KEYWORDS.items():
for keyword in target_keywords:
if re.search(keyword, text, re.IGNORECASE):
return LABELS[label]
return ABSTAIN
#return False to make empty result instead of ABSTAIN
```
4. Click **Predict labels**.
Data programming extension
5. You can now see the final answer from the labeling functions for your targeted question and you can just review it.
Labeled Final Answer
6. You can also see the IAA result in the **Manage functions** dialog. If the score is above 80%, it can be categorized as a good agreement level.
