Sandbox

Sandbox is a platform designed for your LLM experimentation. It allows you to integrate your knowledge base and deploy models easily.

Overview

Sandbox is a key feature within LLM Labs, that provides a user-friendly environment specifically designed for LLM experimentation. It allows you to:

• Connect your preferred base models: Integrate your choice of base models to explore their functionalities.

• Create a dedicated sandbox: Set up a personalized workspace for your LLM experimentation.

• Configure your model: This configuration process defines your models. It involves defining elements like:

  • Instruction: Craft an instruction that specifies the format for user prompts sent to your model. This ensures consistency and clarity in user interactions.

  • Context (knowledge base): Optionally, integrate a context knowledge base to provide additional background information to the model, potentially improving its understanding and response accuracy.

  • Configuration: Fine-tune various parameters for the connected models, such as temperature or token settings, to optimize its performance for your specific use case.

• Run prompts: Test your models with various prompts to evaluate their responses and refine your approach.

Sandbox offers several advantages for LLM enthusiasts and developers:

• Reduced risk: Experiment with different models without committing to deployment, minimizing potential risks associated with real-world use cases.

• Enhanced understanding: Gain deeper insights into individual model and their capabilities through hands-on experimentation.

• Optimized configuration: Fine-tune model parameters within the sandbox to achieve the best possible results for your specific needs.

• Streamlined development: Test and refine your models in a controlled environment before deployment, ensuring optimal performance.

Quick guide

Sandbox is designed for ease of use. Here's a quick guide to get you started:

Step 1: Create your Sandbox

• Click Create new sandbox.

• You can rename your sandbox to make it easier to identify.

Step 2: Configure the model

• In the model you want to configure, click Change model settings from the three-dot menu.

• You can adjust various parameters for your models, such as temperature or token settings.

• Experiment with different configurations to observe their impact on the base model's responses. Learn more about base models.

Step 3: Run prompts

• Enter your desired prompts within the designated area.

• This prompt can be a question, a task instruction, or any text input you want the model to process.

• Click Run selected to to trigger the models to generate responses to your prompts.

Deploy models

Learn more about deploying your models on this page.

Save to library

The Save to library feature allows you to save models from a sandbox. This enables you to reuse these models across various features, such as evaluation, without needing to rebuild them from scratch.

Save to library feature can be used by users within the same workspace, meaning you can share your models with other team members in that workspace.

Save a model

1. To save your model, click the Save to library option from the model's three-dot menu.

   
2. After clicking Save to library, give your model a descriptive name, and click Save model to save it to the library.

   

Access the saved models in Sandbox

1. In the Models section, click Select from library from the three-dot menu.

   
2. Choose the saved model that you want to use, and click Select model.

   

Access the saved models in Evaluation

1. Navigate to the Evaluation page under LLM Labs menu.

2. Click Create evaluation project and choose Automated evaluation project type.

   
3. In the Model dropdown, select the saved models you want to evaluate.

   
4. Continue the creation process for automated evaluation. Learn more on how to create an automated evaluation project.

Add prompts from dataset

In Sandbox, you can efficiently test your models against multiple inputs by importing prompts directly from an existing dataset.

Before you can use this feature, ensure that you have at least one dataset available in your LLM Labs project. Learn more on how to create a dataset.

To add prompts from dataset:

1. In the Prompts section, click Add prompts from dataset from the three-dot menu.

   
2. After selecting this option, you will be able to choose a dataset and map its columns to be used as prompts.

Model configuration

Basic hyperparameters

You can configure your models with your desired settings. Several parameters that you can adjust based on your needs are:

1. Temperature: This parameter controls the randomness of the generated text. A higher temperature will result in more diverse and creative responses, while a lower temperature will produce more focused and predictable responses.

2. Top P: This parameter, also known as nucleus sampling, controls the cumulative probability threshold for token selection. A higher Top P value will result in more focused and relevant responses, while a lower Top P value will allow for more diverse and unexpected responses.

3. Maximum output tokens: This parameter sets the maximum number of tokens that will be generated in the response. A higher maximum length will allow for longer and more detailed responses, while a lower maximum length will result in shorter and more concise responses.

4. Maximum knowledge base tokens: This parameter, defines the upper limit on the number of tokens that can be stored in the knowledge base. This is crucial for maintaining efficient storage and retrieval times. It ensures that the knowledge base doesn't become overloaded with data, which can slow down queries and decrease performance.

5. Similarity score: This parameter controls how closely the generated text matches the original prompt in terms of content and style. A higher similarity score will result in responses that more closely align with the prompt, while a lower similarity score will allow for more divergent responses.

Advanced hyperparameters

Advanced hyperparameters in Sandbox lets you customize your model behavior further by adding additional parameters in JSON format. This gives you greater control over the underlying model's response generation and can optimize performance for specific use cases.

To configure advanced hyperparameters:

1. In the model you want to configure, click Change model settings from the thrree-dot menu.

   
2. Scroll down to the Advanced hyperparameters accordion.

   
3. Enter your desired hyperparameters in valid JSON format. For example:

{ "stream": true }

Anonymize PII

The Anonymize PII feature enhances the privacy and security of sensitive data during inferencing in the Sandbox. This feature is designed to automatically detect and mask Personally Identifiable Information (PII), ensuring that sensitive details remain confidential while retaining readability and context in the output.

How it works

When users submit prompts containing PII in the Sandbox, the Anonymize PII feature securely masks the sensitive information in a structured format. For instance, names, organizations, and locations are replaced with easily identifiable placeholders. These placeholders make it clear that data has been redacted without losing the context required for meaningful results.

Example:

• Original input:

  John Doe works at Datasaur Software and attends meetings with Innova Aspire Inc.

• Masked output:

  [PERSON#0] works at [ORGANIZATION#0] and attends meetings with [ORGANIZATION#1].

This ensures consistency, especially when the same PII subject appears multiple times in the same prompt. The system assigns unique identifiers for different entities, avoiding confusion caused by generic replacements.

How to use

1. In the model you want to configure, click Change model settings from the three-dot menu.

2. Enable Anonymize PII (English only).

   
3. Run your prompt as usual, and all PII in the output will be automatically masked.

   

Auto-generate instruction (BETA)

Auto-generate instruction feature in LLM Labs is designed to help you in generating better-structured system instructions in the Sandbox. This tool enables you to generate customized instructions for your models, making it easier to define the assistant's behavior and optimize responses.

• This feature is currently in BETA and uses OpenAI GPT-4o for instruction generation, providing robust assistance tailored to various models.

• Users are encouraged to refine and double check generated instructions to meet specific needs, as auto-generated content may sometimes require minor adjustments for optimal results.

To auto-generate instructions:

1. Click the Help me write button in a model.

   
2. In the Auto-generate instruction dialog, describe what you’d like the model to achieve.

   

You may also select from example instructions such as Write, debug, and optimize my code to guide the auto-generation process.

1. Click Generate instruction button to receive a structured system instruction based on your input. Review the generated instruction to ensure it matches your intended use case.

   
2. Click the Use this instruction button to add the system instruction into your model.

Cost prediction calculation

Cost prediction provides transparency and predictability in costs. You can see how much your inference activities will cost in LLM Labs.

To see cost prediction:

1. Open your sandbox.

2. Write your prompts, and the predicted cost for each prompt will appear.

   

The cost prediction will calculate the cost based on your available models. The more models you have, the higher the cost will be.

To see the prediction details, click the View predicted cost option from the three-dot menu of each prompt.

It will show you a dialog of the detailed cost. You can also break down and compare the cost prediction based on your available models.