Labeling Function Analysis
Allows users to view the results of their labeling functions, including coverage, overlaps, and conflicts, and to improve performance by training the label model
Labeling function analysis is available only after you generate predictions using labeling functions. You can open the analysis in two ways:
Click Labeling functions, then go to the Analysis tab
Click See labeling function analysis after running predictions
If predictions have not been generated, the analysis page will show an empty state.
Metrics
After running predictions, the following metrics are available:
Coverage: The proportion of data labeled by each labeling function.
Overlap: The proportion of data labeled by multiple labeling functions.
Conflicts: The proportion of data where labeling functions assign different labels.
If you update or add new labeling functions, run Predict labels again to refresh the analysis.
Improve labeling function performance
A good setup typically has high coverage, high overlap, and low conflicts.
Scenario 1: Fairly high coverage, high overlap, and high conflicts
This means many data points are labeled, and multiple labeling functions often assign labels to the same data, but they frequently disagree.
How to improve:
We need to train the label model to get the performance value between labeling functions. The performance value of labeling functions could estimate accuracies and correlations between labeling functions since we know some labeling functions could give high or low signals regarding the label.
Scenario 2: Low coverage, high overlap, and high conflicts
This means only a small portion of data is labeled, but existing labeling functions often overlap and disagree.
How to improve:
Add more labeling functions to increase coverage.
Identify labeling functions that cause conflicts and refine or remove them.
Last updated