LLM Labs (beta)

Enable your integration with models from Datasaur LLM Lab

Supported Labeling Types: Span labeling, Row labeling, Document labeling

Easily integrate with models from Datasaur's LLM Labs. If you've already tested and deployed your experiment in the Datasaur LLM Labs Sandbox, we're here to help! Our integration allows you to use your deployed LLM Sandbox from Datasaur LLM Labs to enhance your labeling process.

Creating an LLM Labs Sandbox

To begin using ML-assisted labeling with LLM Labs, you first need to create and deploy a model from Sandbox. You can see this page to learn more.

The output of the LLM Labs Sandbox must be in JSON object format, aligned with the label set defined in your NLP project. This ensures compatibility with regex-based string matching for labeling in your NLP platform.

1. We have label/question set of Category and Suggestion.

2. Therefore, the expected output or result from the LLM Labs Sandbox should be something like this

{
  "Category": ["Minute of Meeting"],
  "Suggestion": ["Move the 'Issues' section to the top of the notes instead of keeping it at the bottom."]
}

Prompt example to generate the correct JSON Object format

To generate a correct response in JSON object format from the LLM, you need to adjust the user instruction prompt so that the response returns in the expected format. Here are some example prompts you can try:

Given the document text, please extract the following information and present it in JSON format as shown below:
- *Category*: The type of text or notes provided. Please choose one from "Minutes of Meeting," "Draft," or "Budget Plan."
- *Suggestion*: A recommendation based on best practices for creating better notes for "Minutes of Meeting," "Draft," or "Budget Plan."

Instructions Summary:
1. Extract and present the information in the specified JSON format.
2. Ensure that all extracted data is accurate and corresponds directly to the content of each document.

Return the value of extracted fields in JSON structure in plain text, following this JSON FORMAT
{
"Category": [list of Category answer],
"Suggestion": [list of Suggestion answer]
}

VERY IMPORTANT
RETURN THE ANSWER WITHOUT ```json
EXTRACT ANSWER THAT PRECISELY WRITTEN IN DOCUMENT CONTEXT

Advance hyperparameters in the LLM Labs Sandbox

Besides directly adjusting the user instruction prompt in LLM Labs, if you are using the OpenAI model, you can also set advanced hyperparameters in the Hyperparameter configurations.

In the LLM Labs Sandbox, where you configure your model application, you can click the gear icon in the application to open the Hyperparameter configurations. From there, you can add advanced hyperparameters following OpenAI supported schema for Structured Outputs.

Hyperparameter configurations modal

Here's the example of the Advanced hyperparameters:

{
  "response_format": {
    "type": "json_schema",
    "json_schema": {
      "name": "question_answers",
      "schema": {
        "type": "object",
        "properties": {
          "Category": {
            "type": "string",
            "enum": [
              "Minute of Meeting",
              "Draft",
              "Budget Plan"
            ],
            "description": "The answer for question 1, which must be one of the predefined answers or null."
          },
          "Suggestion": {
            "type": "string",
            "description": "The answer for question 2, which can be any string or null."
          }
        },
        "required": [
          "Category",
          "Suggestion"
        ],
      }
    }
  }
}

You can focus adjusting the property and required object to match with your label or question set.

Accessing Your Deployed LLM Labs Sandbox in ML Assisted Labelling

Follow these steps to access your deployed model (from Datasaur LLM Labs) on your ML Assisted Labelling:

1. Create a custom project for row labeling, span labeling or document labeling.

2. Click the gear icon from the extension panel on the right to open the Manage extensions dialog.

3. Enable the ML-assisted labeling extension.

4. Once enabled, select LLM Labs as the service provider and you will see the following menu:

   

   ML-assisted labeling with LLM Labs for span labeling

   

   ML-assisted labeling with LLM Labs for row labeling

   

   ML-assisted labeling with LLM Labs for document labeling

   1. Target text: the column(s) your model should use as input.

   2. Target question: the column(s) you wish to answer.

   3. Model: your deployed model from LLM Labs.

   4. Target pages: define specific page(s) you want to extract from a document.

Prediction Process

Click Predict labels to start predicting labels with your deployed model from LLM Labs.

ML-assisted labeling with LLM Labs in span labeling

ML-assisted labeling with LLM Labs in row labeling

ML-assisted labeling with LLM Labs in document labeling