Inter-Annotator Agreement for Data Programming

Data Programming Inter-Annotator between Labeling Functions

Data programming empowers you to calculate the performance of your models and final answers using inter-annotator agreement (IAA) calculation.

To learn more about inter-annotator agreement, please visit the following link.

Below are the steps to evaluate the performance.

1. Create a project

2. Activate Data programming through the Manage extensions dialog.

   

   Activate Data Programming
3. Create labeling functions for the selected question. You need a minimum of two labeling functions to obtain the IAA value.

   

   Data Programming Editor

Special notes: If you use prelabeled columns as the representative of your models, you can create the labeling function like below :

• If using Snorkel provider, please use this code

  @labeling_function()
  def labeling_function(x) -> int:
    # Implement your logic here
    text = x.columns[x.column_name_to_index['column_name']]
    for key, value in LABELS.items():
      if re.search(key, text, re.IGNORECASE):
        return value
  
    return ABSTAIN

• If using Stegosaurus provider, please use this code and activate Multiple-label template:

  @target_label()
  ABSTAIN = -1
  def label_function(sample):
    text = sample['column_name']
    # text = sample[COLUMN_NAME] if only want to use content from certain column
  
    # Implement your logic here
    # Keywords value on the certain column
    DICT_KEYWORDS = {
      'positive' : ['positive'],
      'negative' : ['negative']
    }
    for label, target_keywords in DICT_KEYWORDS.items():
      for keyword in target_keywords:
        if re.search(keyword, text, re.IGNORECASE):
          return LABELS[label]
    return ABSTAIN
    #return False to make empty result instead of ABSTAIN

1. Click Predict labels.

   

   Data programming extension
2. You can now see the final answer from the labeling functions for your targeted question and you can just review it.

   

   Labeled Final Answer
3. You can also see the IAA result in the Manage functions dialog. If the score is above 80%, it can be categorized as a good agreement level.

   

   Inter-annotator agreement tab