> For the complete documentation index, see [llms.txt](https://docs.datasaur.ai/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.datasaur.ai/llm-projects/models/custom-model.md).
# Custom model
Integrate your own LLMs to Datasaur with custom models, and use them for exploration in Sandbox or for evaluation.
## Integration
While Datasaur offers direct integrations with some providers, you can also connect to models hosted on various third-party platforms or your own infrastructure using the **Custom model** feature.
{% hint style="info" %}
To simplify integration, Datasaur's custom model connection adheres to the widely adopted API structure defined by OpenAI for its completions or chat completions endpoints.
This means that if your self-hosted model or third-party serving framework exposes an endpoint that mimics the OpenAI API format, connecting it to Datasaur is straightforward.
{% endhint %}
### API specification
To simplify integration, Datasaur connects to custom models through an API structure compatible with the OpenAI Chat Completions API.
Think of it as using a standard interface: if your model serving framework (such as TGI) exposes an OpenAI-compatible API, Datasaur can connect to it directly with minimal configuration.
### How the API connection works
{% hint style="info" %}
Please note that the `streaming` option is currently disabled for custom models.
{% endhint %}
When Datasaur uses your custom model, here’s what happens:
#### 1. Datasaur sends a request
Datasaur sends information to your model's address using a standard web method (`POST`). This request goes to a specific path, usually `/v1/chat/completions`, added to the main address you provide. The request includes:
* **Headers:** Tells the server the data is in JSON format. If you added an API Key in Datasaur, it sends that key for security (`Authorization: Bearer YOUR_API_KEY`).
* **JSON body:** The request body contains:
* `model`: The name of the specific model you want to use.
* `messages`: The conversation history, including instructions ("system" message) and the user's input ("user" message).
* Optional settings like `temperature` (for creativity) or `max_tokens` (to limit response length).
Example request data:
```json
{
"model": "tgi",
"messages": [
{
"role": "system",
"content": "You are a helpful assistant."
},
{
"role": "user",
"content": "Explain what an LLM is in simple terms."
}
],
"temperature": 0.7,
"max_tokens": 150
}
```
#### 2. Your model sends a response
If the request succeeds, your model server returns a `200 OK` response with a JSON payload. The response typically includes:
* `id`: A unique ID for this conversation turn.
* `model`: The model used to generate the response.
* `choices`: An array (usually just one item) containing the model's reply:
* `message`: The actual text generated by the model (`content`) and its role (`assistant`).
* `usage` (Optional): Token usage statistics.
Example response data:
{% code overflow="wrap" %}
```json
{
"id": "chatcmpl-randomid12345",
"object": "chat.completion",
"created": 1713867000, // Timestamp
"model": "your-model-name",
"choices": [
{
"index": 0,
"message": {
"role": "assistant",
"content": "An LLM, or Large Language Model, is like a super-smart computer program that's read tons of text, so it can understand and write text almost like a human!"
},
"finish_reason": "stop" // Why the model stopped writing
}
],
"usage": {
"prompt_tokens": 30,
"completion_tokens": 55,
"total_tokens": 85
}
}
```
{% endcode %}
### Text Generation Inference (TGI) & Hugging Face
* **Main Address (Base URL):** Enter the address where your TGI server is running. This might look like `http://your-tgi-server-ip:8080/v1` or maybe use port `80` if using certain hosting like Hugging Face Inference Endpoints. Add `/v1` at the end.
* **Model Name (in request data):** TGI usually runs one main model at a time. You might just need to put `"tgi"` as the model name, or use the specific Hugging Face name the model was started with (like `"NousResearch/Nous-Hermes-2"`). Check your TGI setup. [Learn more about TGI](https://huggingface.co/docs/text-generation-inference/en/basic_tutorials/consuming_tgi#curl).
* **API Key:** If you're using TGI, especially through services like Hugging Face, you'll likely need an API key. Get this key from your TGI provider (like your Hugging Face Access Token), and put it in the API Key field in Datasaur's custom model settings.
## Connect custom models
To connect a custom model:
1. Navigate to the **Models catalog** page, then click **Manage providers**.
2. Select **Custom model**.
3. Input your model credentials. The required credentials are:
1. **Endpoint URL**: The endpoint URL of your model.
2. **API key**: The API key of your model.
3. **Model name**: Your desired model name to be used in LLM Labs.
{% hint style="info" %}
If you are adding a custom model from an LLM provider like Hugging Face, you only need to input the endpoint URL without the `/v1/chat/completions` suffix.
{% endhint %}
5. Once you’ve added your credentials, click the **Add custom model** button, and your custom model will be available in LLM Labs.
Models available
## Manage custom models
To manage your custom models, click the three-dot menu on the model card. From there, you can:
* View details.
* Try in Sandbox.
* Edit.
* Delete.
### Try in Sandbox
Click **Try in Sandbox,** and you'll be automatically taken to a Sandbox. This allows you to use it as a base model and test how it works with additional instructions and various prompts. Learn more about [Sandbox](/llm-projects/sandbox.md).
Try in Sandbox
### Edit custom model
Click **Edit** to modify the endpoint URL, API key, and model name. Once you've updated the model credentials, click **Save custom model** to save your changes.
### Delete custom model
Click **Delete** to delete your model from LLM Labs. Confirm the deletion by clicking the **Delete custom model** button.
Delete custom model
---
# Agent Instructions
This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com.
## Querying This Documentation
If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question.
Perform an HTTP GET request on the current page URL with the `ask` query parameter, and the optional `goal` query parameter:
```
GET https://docs.datasaur.ai/llm-projects/models/custom-model.md?ask=&goal=
```
`ask` is the immediate question: it should be specific, self-contained, and written in natural language.
`goal` is optional and describes the broader end goal you are ultimately trying to accomplish on behalf of the user. GitBook uses it to tailor the answer towards what is most useful for that goal.
The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.
Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.