Datasaur
Visit our websitePricingBlogPlaygroundAPI Docs
  • Welcome to Datasaur
    • Getting started with Datasaur
  • Data Studio Projects
    • Labeling Task Types
      • Span Based
        • OCR Labeling
        • Audio Project
      • Row Based
      • Document Based
      • Bounding Box
      • Conversational
      • Mixed Labeling
      • Project Templates
        • Test Project
    • Creating a Project
      • Data Formats
      • Data Samples
      • Split Files
      • Consensus
      • Dynamic Review Capabilities
    • Pre-Labeled Project
    • Let's Get Labeling!
      • Span Based
        • Span + Line Labeling
      • Row & Document Based
      • Bounding Box Labeling
      • Conversational Labeling
      • Label Sets / Question Sets
        • Dynamic Question Set
      • Multiple Label Sets
    • Reviewing Projects
      • Review Sampling
    • Adding Documents to an Ongoing Project
    • Export Project
  • LLM Projects
    • LLM Labs Introduction
    • Sandbox
      • Direct Access LLMs
      • File Attachment
      • Conversational Prompt
    • Deployment
      • Deployment API
    • Knowledge base
      • External Object Storage
      • File Properties
      • Chunk Editor
    • Models
      • Amazon SageMaker JumpStart
      • Amazon Bedrock
      • Open AI
      • Azure OpenAI
      • Vertex AI
      • Custom model
      • Fine-tuning
      • LLM Comparison Table
    • Evaluation
      • Automated Evaluation
        • Multi-application evaluation
        • Custom metrics
      • Ranking (RLHF)
      • Rating
      • Performance Monitoring
    • Dataset
    • Pricing Plan
  • Workspace Management
    • Workspace
    • Role & Permission
    • Analytics
      • Inter-Annotator Agreement (IAA)
        • Cohen's Kappa Calculation
        • Krippendorff's Alpha Calculation
      • Custom Report Builder
      • Project Report
      • Evaluation Metrics
    • Activity
    • File Transformer
      • Import Transformer
      • Export Transformer
      • Upload File Transformer
      • Running File Transformer
    • Label Management
      • Label Set Management
      • Question Set Management
    • Project Management
      • Self-Assignment
        • Self-Unassign
      • Transfer Assignment Ownership
      • Reset Labeling Work
      • Mark Document as Complete
      • Project Status Workflow
        • Read-only Mode
      • Comment Feature
      • Archive Project
    • Automation
      • Action: Create Projects
  • Assisted Labeling
    • ML Assisted Labeling
      • Amazon Comprehend
      • Amazon SageMaker
      • Azure ML
      • CoreNLP NER
      • CoreNLP POS
      • Custom API
      • FewNERD
      • Google Vertex AI
      • Hugging Face
      • LLM Assisted Labeling
        • Prompt Examples
        • Custom Provider
      • LLM Labs (beta)
      • NLTK
      • Sentiment Analysis
      • spaCy
      • SparkNLP NER
      • SparkNLP POS
    • Data Programming
      • Example of Labeling Functions
      • Labeling Function Analysis
      • Inter-Annotator Agreement for Data Programming
    • Predictive Labeling
  • Assisted Review
    • Label Error Detection
  • Building Your Own Model
    • Datasaur Dinamic
      • Datasaur Dinamic with Hugging Face
      • Datasaur Dinamic with Amazon SageMaker Autopilot
  • Advanced
    • Script-Generated Question
    • Shortcuts
    • Extensions
      • Labels
      • Review
      • Document and Row Labeling
      • Bounding Box Labels
      • List of Files
      • Comments
      • Analytics
      • Dictionary
      • Search
      • Labeling Guidelines
      • Metadata
      • Grammar Checker
      • ML Assisted Labeling
      • Data Programming
      • Datasaur Dinamic
      • Predictive Labeling
      • Label Error Detection
      • LLM Sandbox
    • Tokenizers
  • Integrations
    • External Object Storage
      • AWS S3
        • With IRSA
      • Google Cloud Storage
      • Azure Blob Storage
      • Dropbox
    • SAML
      • Okta
      • Microsoft Entra ID
    • SCIM
      • Okta
      • Microsoft Entra ID
    • Webhook Notifications
      • Webhook Signature
      • Events
      • Custom Headers
    • Robosaur
      • Commands
        • Create Projects
        • Apply Project Tags
        • Export Projects
        • Generate Time Per Task Report
        • Split Document
      • Storage Options
  • API
    • Datasaur APIs
    • Credentials
    • Create Project
      • New mutation (createProject)
      • Python Script Example
    • Adding Documents
    • Labeling
      • Create Label Set
      • Add Label Sets into Existing Project
      • Get List of Label Sets in a Project
      • Add Label Set Item into Project's Label Set
      • Programmatic API Labeling
      • Inserting Span and Arrow Label into Document
    • Export Project
      • Custom Webhook
    • Get Data
      • Get List of Projects
      • Get Document Information
      • Get List of Tags
      • Get Cabinet
      • Export Team Overview
      • Check Job
    • Custom OCR
      • Importable Format
    • Custom ASR
    • Run ML-Assisted Labeling
  • Security and Compliance
    • Security and Compliance
      • 2FA
  • Compatibility & Updates
    • Common Terminology
    • Recommended Machine Specifications
    • Supported Formats
    • Supported Languages
    • Release Notes
      • Version 6
        • 6.114.0
        • 6.113.0
        • 6.112.0
        • 6.111.0
        • 6.110.0
        • 6.109.0
        • 6.108.0
        • 6.107.0
        • 6.106.0
        • 6.105.0
        • 6.104.0
        • 6.103.0
        • 6.102.0
        • 6.101.0
        • 6.100.0
        • 6.99.0
        • 6.98.0
        • 6.97.0
        • 6.96.0
        • 6.95.0
        • 6.94.0
        • 6.93.0
        • 6.92.0
        • 6.91.0
        • 6.90.0
        • 6.89.0
        • 6.88.0
        • 6.87.0
        • 6.86.0
        • 6.85.0
        • 6.84.0
        • 6.83.0
        • 6.82.0
        • 6.81.0
        • 6.80.0
        • 6.79.0
        • 6.78.0
        • 6.77.0
        • 6.76.0
        • 6.75.0
        • 6.74.0
        • 6.73.0
        • 6.72.0
        • 6.71.0
        • 6.70.0
        • 6.69.0
        • 6.68.0
        • 6.67.0
        • 6.66.0
        • 6.65.0
        • 6.64.0
        • 6.63.0
        • 6.62.0
        • 6.61.0
        • 6.60.0
        • 6.59.0
        • 6.58.0
        • 6.57.0
        • 6.56.0
        • 6.55.0
        • 6.54.0
        • 6.53.0
        • 6.52.0
        • 6.51.0
        • 6.50.0
        • 6.49.0
        • 6.48.0
        • 6.47.0
        • 6.46.0
        • 6.45.0
        • 6.44.0
        • 6.43.0
        • 6.42.0
        • 6.41.0
        • 6.40.0
        • 6.39.0
        • 6.38.0
        • 6.37.0
        • 6.36.0
        • 6.35.0
        • 6.34.0
        • 6.33.0
        • 6.32.0
        • 6.31.0
        • 6.30.0
        • 6.29.0
        • 6.28.0
        • 6.27.0
        • 6.26.0
        • 6.25.0
        • 6.24.0
        • 6.23.0
        • 6.22.0
        • 6.21.0
        • 6.20.0
        • 6.19.0
        • 6.18.0
        • 6.17.0
        • 6.16.0
        • 6.15.0
        • 6.14.0
        • 6.13.0
        • 6.12.0
        • 6.11.0
        • 6.10.0
        • 6.9.0
        • 6.8.0
        • 6.7.0
        • 6.6.0
        • 6.5.0
        • 6.4.0
        • 6.3.0
        • 6.2.0
        • 6.1.0
        • 6.0.0
      • Version 5
        • 5.63.0
        • 5.62.0
        • 5.61.0
        • 5.60.0
  • Deployment
    • Self-Hosted
      • AWS Marketplace
        • Data Studio
        • LLM Labs
Powered by GitBook
On this page
  • Overview
  • Integration
  • Connect your custom model
  • Managing custom model
  1. LLM Projects
  2. Models

Custom model

Overview

LLM Labs Custom Models allows you to integrate your own large language models (LLMs) from various providers into Datasaur's LLM Labs. This feature empowers you to leverage your specialized models alongside Datasaur's built-in capabilities.

Integration

While Datasaur offers direct integrations with some providers, you can also connect to models hosted on various third-party platforms or your own infrastructure using the "Custom model" feature.

To simplify integration, Datasaur's Custom Model connection adheres to the widely adopted API structure defined by OpenAI for its completions or chat completions endpoints. This means that if your self-hosted model or third-party serving framework exposes an endpoint that mimics the OpenAI API format, connecting it to Datasaur is straightforward.

API specification

To make things easy, Datasaur connects to these custom models using a common method – an API structure that looks just like the one used by OpenAI (specifically, the "Chat Completions" API).

Think of it like using a standard plug: if your model hosting tool (like TGI) provides this standard API interface (an OpenAI-compatible API), Datasaur can plug right in.

How the API connection works

Please note that the streaming option is currently disabled for custom models.

When Datasaur uses your custom model, here’s basically what happens:

  1. Datasaur Sends a Request: Datasaur sends information to your model's address using a standard web method (POST). This request goes to a specific path, usually /v1/chat/completions, added to the main address you provide. The request includes:

    • Extra Info (Headers): Tells the server the data is in JSON format. If you added an API Key in Datasaur, it sends that key for security (Authorization: Bearer YOUR_API_KEY).

    • The Actual Data (JSON Body): This contains the important bits:

      • model: The name of the specific model you want to use.

      • messages: The conversation history, including instructions ("system" message) and the user's input ("user" message).

      • Optional settings like temperature (for creativity) or max_tokens (to limit response length).

    Example Request Data:

    {
      "model": "tgi",
      "messages": [
        {
          "role": "system",
          "content": "You are a helpful assistant."
        },
        {
          "role": "user",
          "content": "Explain what an LLM is in simple terms."
        }
      ],
      "temperature": 0.7,
      "max_tokens": 150
    }
  2. Your Model Sends a Response: If everything works, your model hosting tool sends back a success message (200 OK) with its own JSON data, including:

    • id: A unique ID for this conversation turn.

    • model: Which model actually answered.

    • choices: An array (usually just one item) containing the model's reply:

      • message: The actual text generated by the model (content) and its role (assistant).

    • usage (Optional): Information about how many "pieces" of text (tokens) were used for the prompt and the answer.

    Example Response Data:

    {
      "id": "chatcmpl-randomid12345",
      "object": "chat.completion",
      "created": 1713867000, // Timestamp
      "model": "your-model-name",
      "choices": [
        {
          "index": 0,
          "message": {
            "role": "assistant",
            "content": "An LLM, or Large Language Model, is like a super-smart computer program that's read tons of text, so it can understand and write text almost like a human!"
          },
          "finish_reason": "stop" // Why the model stopped writing
        }
      ],
      "usage": {
        "prompt_tokens": 30,
        "completion_tokens": 55,
        "total_tokens": 85
      }
    }
    

Text Generation Inference (TGI) & Huggingface

  • Main Address (Base URL): Put the address where your TGI server is running. This might look like http://your-tgi-server-ip:8080/v1 or maybe use port 80 if using certain hosting like Hugging Face Inference Endpoints. Add /v1 at the end.

  • API Key: If you're using TGI, especially through services like Hugging Face, you'll likely need an API Key or Token. Get this key from your TGI provider (like your Hugging Face Access Token) and put it in the API Key field in Datasaur's Custom Model settings.

Connect your custom model

To connect to a custom model:

  1. Navigate to the Models page.

  2. Click the Manage providers button.

  3. Click the Custom model button.

  4. Input your model credentials. The required credentials are:

    1. Endpoint URL: The endpoint URL of your model.

    2. API key: The API key of your model.

    3. Model name: Your desired model name to be used in LLM Labs.

If you are adding a custom model from an LLM provider like Hugging Face, you only need to input the endpoint URL without the /v1/chat/completions suffix.

  1. Once you’ve added your credentials, click the Add custom model button, and your custom model will be available in LLM Labs.

Managing custom model

To manage your custom model, click the three-dots icon on the model card.

From there, you can:

  1. View Details

  2. Try in Sandbox

  3. Edit

  4. Delete

View details

Once you click the View details menu, you will see the endpoint name of the model displayed in the details dialog.

Try in Sandbox

Edit custom model

Once you click on the Edit menu, you will be able to modify the endpoint name, API key, and model name. Once you've updated the model credentials, click the Edit custom model button to save your changes.

Delete custom model

Once you click on the Delete menu, you will be able to delete your model from LLM Labs. Confirm the deletion by clicking the Delete custom model button.

Last updated 25 days ago

Model Name (in Request Data): TGI usually runs one main model at a time. You might just need to put "tgi" as the model name, or use the specific Hugging Face name the model was started with (like "NousResearch/Nous-Hermes-2"). Check your TGI setup. .

Once you click the Try in Sandbox menu, you'll be automatically taken to a Sandbox project. This allows you to integrate and test the model within your specific application. .

Learn more about TGI
Learn more about Sandbox
Manage providers
Custom models option
Add custom model dialog
Models available
View details
Try in Sandbox
Edit model
Delete custom model