Inter-Annotator Agreement for Data Programming

Data Programming Inter-Annotator between Labeling Functions

Data programming empowers you to calculate the performance of your models and final answers using Inter-Annotator Agreement calculation.

To learn more about Inter-Annotator Agreement, please visit the following link.

Below are the steps to evaluate the performance.

  1. Create a project

  2. Activate Data programming through extension manager.

  3. Create labeling functions for the selected question. You need a minimum of two labeling functions to obtain the IAA value.

Special notes: If you use prelabeled columns as the representative of your models, you can create the labeling function like below :

  • If using Snorkel provider, please use this code

    def labeling_function(x) -> int:
      # Implement your logic here
      text = x.columns[x.column_name_to_index['column_name']]
      for key, value in LABELS.items():
        if, text, re.IGNORECASE):
          return value
      return ABSTAIN
  • If using Stegosaurus provider, please use this code and activate "Multiple label template":

    ABSTAIN = -1
    def label_function(sample):
      text = sample['column_name']
      # text = sample[COLUMN_NAME] if only want to use content from certain column
      # Implement your logic here
      # Keywords value on the certain column
        'positive' : ['positive'],
        'negative' : ['negative']
      for label, target_keywords in DICT_KEYWORDS.items():
        for keyword in target_keywords:
          if, text, re.IGNORECASE):
            return LABELS[label]
      return ABSTAIN
      #return False to make empty result instead of ABSTAIN
  1. Click “Predict labels”.

  2. You can now see the final answer from the labeling functions for your targeted question and you can just review it.

  3. You can also see the IAA performance result in Data Programming window. If the score is above 80%, it can be categorized as a good agreement level.

Last updated