# Inter-Annotator Agreement for Data Programming
**Data programming** evaluates the performance of labeling functions using [inter-annotator agreement](/workspace-management/analytics/inter-annotator-agreement.md) (IAA).
## Steps
To evaluate the performance of labeling functions:
1. Activate **Data programming** through the **Manage extensions** dialog.
2. Create labeling functions for the selected question. At least two labeling functions are required to calculate IAA.
3. Click **Predict labels**. You can now view the final labels generated by the labeling functions for your selected question.
You can review the predicted labels and view the IAA score in the **Manage Functions** dialog. An IAA score above 80% indicates good agreement.
## Pre-labeled columns as model representatives
If you use pre-labeled columns as representatives of your models, you can create labeling functions based on those values.
* If using Snorkel provider, please use this code
```python
@labeling_function()
def labeling_function(x) -> int:
# Implement your logic here
text = x.columns[x.column_name_to_index['column_name']]
for key, value in LABELS.items():
if re.search(key, text, re.IGNORECASE):
return value
return ABSTAIN
```
* If using Stegosaurus provider, please use this code and activate **Multiple-label template**:
```python
@target_label()
ABSTAIN = -1
def label_function(sample):
text = sample['column_name']
# text = sample[COLUMN_NAME] if only want to use content from certain column
# Implement your logic here
# Keywords value on the certain column
DICT_KEYWORDS = {
'positive' : ['positive'],
'negative' : ['negative']
}
for label, target_keywords in DICT_KEYWORDS.items():
for keyword in target_keywords:
if re.search(keyword, text, re.IGNORECASE):
return LABELS[label]
return ABSTAIN
#return False to make empty result instead of ABSTAIN
```
---
# Agent Instructions: Querying This Documentation
If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question.
Perform an HTTP GET request on the current page URL with the `ask` query parameter:
```
GET https://docs.datasaur.ai/assisted-labeling/data-programming/inter-annotator-agreement-for-data-programming.md?ask=
```
The question should be specific, self-contained, and written in natural language.
The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.
Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.