Sandbox

Overview

Sandbox is a key feature within LLM Labs, provides a user-friendly environment specifically designed for LLM experimentation. It allows you to:

• Connect your preferred LLM model: Integrate your choice of LLM models to explore their functionalities.

• Create a dedicated sandbox: Set up a personalized workspace for your LLM experimentation.

• Configure your application: This configuration process defines your LLM application. It involves defining elements like:

  • Instruction: Craft an instruction that specifies the format for user prompts sent to your LLM model. This ensures consistency and clarity in user interactions.

  • Context (Knowledge base): Optionally, integrate a context knowledge base to provide additional background information to the LLM model, potentially improving its understanding and response accuracy.

  • Configuration: Fine-tune various parameters for the connected LLM model, such as temperature or token settings, to optimize its performance for your specific use case.

• Run prompts: Test your LLM models with various prompts to evaluate their responses and refine your approach.

Sandbox offers several advantages for LLM enthusiasts and developers:

• Reduced Risk: Experiment with different models without committing to deployment, minimizing potential risks associated with real-world use cases.

• Enhanced Understanding: Gain deeper insights into individual LLM models and their capabilities through hands-on experimentation.

• Optimized Configuration: Fine-tune model parameters within the sandbox to achieve the best possible results for your specific needs.

• Streamlined Development: Test and refine your LLM applications in a controlled environment before deployment, ensuring optimal performance.

Getting Started with the Sandbox

Sandbox is designed for ease of use. Here's a quick guide to get you started:

Step 1: Create Your Sandbox

• Click on Create new sandbox to establish your dedicated workspace.

• You can assign a descriptive name to your sandbox for easy identification.

Step 2: Configure the sandbox application

• Access the Configuration section within your sandbox.

• You can adjust various parameters for the connected LLM model, such as temperature or token settings.

• Experiment with different configurations to observe their impact on the model's responses. Learn More about Models.

Step 3: Run Prompts

• Enter your desired prompt within the designated area.

• This prompt can be a question, a task instruction, or any text input you want the LLM model to process.

• Click Run all to trigger the model's response based on your prompt.

Deploying the LLM

Save to library

The Save to library feature allows you to save applications from the sandbox into a library. This enables you to reuse these applications across various features, such as evaluation, without needing to rebuild them from scratch.

How to save application to library

To save your application to library, you can just simply click the Save to library option from the application three-dots icon.

Once you’ve clicked the Save to library button, you will need to enter the application name that you want to save in the library.

Click the Save application button once you’ve entered the name. And the application will be saved in the library.

Access the application from library in sandbox

1. Navigate to the Sandbox page.

2. Click the arrow button next to the Add application button.

   
3. Choose the saved application that you wish to use, and click Select application button to use the saved application.

   

Access the application from library in evaluation

1. Navigate to the Evaluation page under LLM Labs menu.

2. Click the Create evaluation project button and choose Automated evaluation project type.

   
3. Once you’ve in the creation wizard, choose the application that is saved from the library.

   
4. Continue the creation wizard process for automated evaluation. Learn more on how to create an automated evaluation project.

Add prompts from dataset

In LLM Labs, you can efficiently test your application against multiple inputs by importing prompts directly from an existing dataset.

How to add prompts from a dataset

1. Navigate to the Sandbox page within your project.

2. On the right side of the screen, locate the prompt panel.

3. Click the the dropdown menu beside the + Add prompt button.

4. From the menu, click the Add prompts from dataset button.

   

After selecting this option, you will be able to choose a dataset and map its columns to be used as prompts.

Application Configuration

You can configure your application with your desired settings. Several parameters that you can adjust based on your needs are:

1. Temperature: This parameter controls the randomness of the generated text. A higher temperature will result in more diverse and creative responses, while a lower temperature will produce more focused and predictable responses.

2. Top P: This parameter, also known as nucleus sampling, controls the cumulative probability threshold for token selection. A higher Top P value will result in more focused and relevant responses, while a lower Top P value will allow for more diverse and unexpected responses.

3. Maximum output tokens: This parameter sets the maximum number of tokens that will be generated in the response. A higher maximum length will allow for longer and more detailed responses, while a lower maximum length will result in shorter and more concise responses.

4. Maximum knowledge base tokens: This parameter, defines the upper limit on the number of tokens that can be stored in the knowledge base. This is crucial for maintaining efficient storage and retrieval times. It ensures that the knowledge base doesn't become overloaded with data, which can slow down queries and decrease performance.

5. Similarity score: This parameter controls how closely the generated text matches the original prompt in terms of content and style. A higher similarity score will result in responses that more closely align with the prompt, while a lower similarity score will allow for more divergent responses.

Advanced hyperparameters

Advanced hyperparameters in LLM Labs Sandbox allow users to customize their model behavior further by adding additional parameters in JSON format. This gives users greater control over the underlying model's response generation and can optimize performance for specific use cases.

Get started

1. Navigate to your application setup within LLM Labs Sandbox.

2. Open the application configuration panel.

   
3. Scroll down to the section labeled Advanced hyperparameters.

   
4. Enter your desired hyperparameters in valid JSON format.

JSON Format Example

{ "stream": true }

Anonymize PII

The Anonymize PII feature enhances the privacy and security of sensitive data during inferencing in the Sandbox. This feature is designed to automatically detect and mask Personally Identifiable Information (PII), ensuring that sensitive details remain confidential while retaining readability and context in the output.

How it works

When users submit prompts containing PII in the Sandbox, the Anonymize PII feature securely masks the sensitive information in a structured format. For instance, names, organizations, and locations are replaced with easily identifiable placeholders. These placeholders make it clear that data has been redacted without losing the context required for meaningful results.

Example:

• Original input: “John Doe works at Datasaur Software and attends meetings with Innova Aspire Inc.”

• Masked output: “[PERSON#0] works at [ORGANIZATION#0] and attends meetings with [ORGANIZATION#1]”

This ensures consistency, especially when the same PII subject appears multiple times in the same prompt. The system assigns unique identifiers for different entities, avoiding confusion caused by generic replacements.

How to use

1. Navigate to the sandbox where you will configure your model application.

2. Open the Hyperparameter configurations dialog by clicking the gear icon in the application.

3. Look for the toggle labeled Anonymize PII (English only) within the configuration options, and simply enable the toggle to activate the feature.

   
4. Run your prompt as usual, and all PII in the output will be automatically masked.

   

Auto-generate instruction (BETA)

Auto generate instruction feature in LLM Labs is designed to help user in generating better-structured system instructions in the Sandbox. This tool enables users to generate customized instructions for LLM applications, making it easier to define the assistant's behavior and optimize responses.

Get started

To auto-generate instructions:

1. Navigate to the Sandbox page under LLM Labs menu, and open your Sandbox.

2. Select your application and click the Auto-generate button.

   
3. In the Auto-generate instruction dialog, describe what you’d like the model to achieve. For example, enter tasks like "Automate professional email drafting" or "Summarize contracts and legal terms".

   

1. Click Generate instruction button to receive a structured system instruction based on your input. Review the generated instruction to ensure it matches your intended use case, making adjustments as necessary for clarity or specificity.

   
2. Click the Use this instruction button to add the system instruction into your application.

   

Cost Prediction Calculation

Overview

The Cost Prediction for Inference feature aims to enhance user experience by providing transparency and predictability in terms of the costs incurred during language model inference. Users can now estimate the financial implications of their inferencing activities within LLM Lab.

How to see the cost prediction?

1. Open your sandbox.

2. Write your prompt query, and the predicted cost from your prompt templates will be shown.

   

View Prediction Details

To see the prediction details, you just have to click the cost prediction. It will show you a dialog of the detailed cost.

You can also break down and compare the cost prediction based on your available application.