Data Programming

Assisted-labeling feature to help you generate label using rules

Introduction

Datasaur's Data programming extension offers an advanced solution for processing large datasets. By leveraging a set of algorithms and heuristics, it automates data labeling—a task typically done manually. This is particularly beneficial for handling huge data volumes, significantly improving labeling efficiency and accuracy. The automation allows users to focus on more critical aspects of data analysis and model training, making it a key component in building a high-quality ML models. You can adjust the labeling functions, define your rules and create a pattern to help you easily label the data.

Key Features

• Labeling functions: The Data programming extension uses various heuristics or rules known as labeling functions. While these functions might not be highly accurate individually, collectively they provide better predictions than random selection. A series of examples of labeling functions can be found here. Labeling functions can be written using the provided template in Python, and some libraries are supported by default.

• Labeling function analysis: View results of labeling functions, including coverage, overlaps, and conflicts. Improve performance by training the label model. Supports both span-based and row-based data. For more details, click here.

• Inter-annotator agreement for labeling functions: Calculate the performance of labeling functions and reviewed answers using inter-annotator agreement. For more details, click here.

Supported Libraries

If you require any additional libraries, please reach out to us by contacting [email protected].

Name	Version
pandas	1.4.4 and later
textblob	0.17.1 and later
nltk	3.7 and later
spacy	3.4.1 and later
scipy	1.9.1
numpy	1.23.3
transformers	4.28.1
requests	2.28.1 and later
datasets	2.7.0
openai	0.27.0
stanza	1.5.0 and later
spacy-fastlang	1.0.1 and later
lxml	4.9.2

Quick Start Guide

How to use Data Programming in General

1. Add Data programming extension from the Manage extensions dialog.

   
2. The Data programming extension will appear on your right. Let's break down what we have here:

   • Target question/Label set: Choose the questions or label set that you want to target for Data programming usage.

   • Multiple-label template: If you turn this on, it will create a labeling function template that can predict based on multiple labels. By default, the labeling function logic only specified to predict 1 label from all defined labels. However, Datasaur also provides a multilabel labeling function template if a user needs labeling function logic that is sufficient for more than 1 label. Please find the template here.

   • Labeling functions: This button will take you to the Manage functions dialog, covering the labeling function settings, labeling function analysis, and inter-annotator agreement.

   • Predict labels: After creating your labeling functions, you can start predicting the answers or labels using those functions.

     

     Data Programming Extension
3. You can create labeling functions by clicking the Labeling functions button. By default it will open the Labeling function settings tab, where you can add your labeling functions. It also provides you with a code template for a labeling function based on your label set. Note: Pay attention to the comment we've included there; you can start editing your logic where we write (Start editing here!)— the previous codes and lines are not supposed to be edited.

   

   Data Programming Code Editor
4. Close the Manage functions dialog and click Predict labels in the Data programming extension.

Labeling Function Template

583B

Default Labeling Function for Row (1).txt

Open

653B

Default Labeling Function for Span (1).txt

Open

775B

Multiple Labeling Function for Row.txt

Open

978B

Multiple Labeling Function for Span.txt

Open

Edit and adjust Labeling Functions

1. First, click Add button to create a labeling function. In the code editor, a template will be generated.

   

   Data Programming Pop Up Window
2. To rename a labeling function, hover over the name and click the edit button. Type the new name, then click the save button to save it, or cancel button to cancel it.

   
3. Removing the labeling function can be done in two ways, delete one by one or delete multiple at once. Select one or multiple labeling functions via the checkbox and click Delete.

   

   Delete multiple
4. There will be a confirmation dialog to confirm the deletion. After you click OK, the selected labeling functions will be deleted.

   
5. Use a toggle which inlines with the labeling function to activate or inactivate the labeling function for prediction.

   

Build Labeling Functions in detail

• By default, labeling_function only provides 1 label, which is defined on this line

  #Assign target label based on LABELS dictionary
  @apply_label(label=LABELS['labelA'])

• By default, labeling_function process text that contains all columns in one row.

  ## please check <link section 'text' in gitbook> for more info
  text = list(sample.values())[0]

  If need to process only one column, then use:

  text = sample[<COLUMN_NAME>]

  If need to process certain columns, then use:

  text = ' '.join([sample[<COLUMN_NAME_A>], sample[<COLUMN_NAME_B>]])

• Labeling Function returns boolean as output for row-based and match_list as output for span-based.

  match_list is a form of a list of match token index (format: [start_`index, end_index])

  In example:

  >>> text = "Russian and American Alien Spaceship Claims Raise Eyebrows"
  >>> target_token = ["Russian", "American"]
  >>> match_list = [[0,7],[12,20]]

  match_list is a list of target_token positions regarding to text

  Special Notes:

  if you use regex in your logic codes, you can find match_list with regex.finditer

  	# *TARGET could be **keyword** or **regex pattern.*** 
  
  >>> match_list = [re.finditer(target, text) for target in TARGET_KEYWORDS]
  
  or
  
  >>> date = re.compile(r"(19|20)\\d\\d[- /.](0[1-9]|[012])[- /.](0[1-9]|[12][0-9]|3[01])")
  >>> PATTERNS = [date]
  >>> match_list = [re.finditer(pattern, text) for pattern in PATTERNS]

• Labeling function remover

  • Row Labeling

    def labeling_function(sample):
        return False

  • Span Labeling

    def labeling_function(sample):
    		
    match_list = []
    return match_list