Inter-Annotator Agreement for Data Programming

Measure agreement between labeling functions to evaluate performance.

Data programming evaluates the performance of labeling functions using inter-annotator agreement (IAA).

Steps

To evaluate the performance of labeling functions:

1. Activate Data programming through the Manage extensions dialog.

   
2. Create labeling functions for the selected question. At least two labeling functions are required to calculate IAA.

   
3. Click Predict labels. You can now view the final labels generated by the labeling functions for your selected question.

   

You can review the predicted labels and view the IAA score in the Manage Functions dialog. An IAA score above 80% indicates good agreement.

Pre-labeled columns as model representatives

If you use pre-labeled columns as representatives of your models, you can create labeling functions based on those values.

• If using Snorkel provider, please use this code

  @labeling_function()
  def labeling_function(x) -> int:
    # Implement your logic here
    text = x.columns[x.column_name_to_index['column_name']]
    for key, value in LABELS.items():
      if re.search(key, text, re.IGNORECASE):
        return value
  
    return ABSTAIN

• If using Stegosaurus provider, please use this code and activate Multiple-label template:

  @target_label()
  ABSTAIN = -1
  def label_function(sample):
    text = sample['column_name']
    # text = sample[COLUMN_NAME] if only want to use content from certain column
  
    # Implement your logic here
    # Keywords value on the certain column
    DICT_KEYWORDS = {
      'positive' : ['positive'],
      'negative' : ['negative']
    }
    for label, target_keywords in DICT_KEYWORDS.items():
      for keyword in target_keywords:
        if re.search(keyword, text, re.IGNORECASE):
          return LABELS[label]
    return ABSTAIN
    #return False to make empty result instead of ABSTAIN