Custom metrics

Overview

Custom metrics in automated evaluation projects let you define your own evaluation setup to fit your project’s needs. With custom metrics, you control the evaluation criteria, scoring ranges, and evaluator models to ensure accurate and meaningful assessments.

Get started

To use custom metrics:

1. Navigate to the Evaluation page under LLM Labs menu.

2. Click the Create evaluation project button and choose Automated evaluation project type, then Continue.

   
3. Configure your evaluation by selecting the models to evaluate and choosing a dataset from the library. If you don’t have one, you can also upload a dataset in a CSV format containing two columns: prompt and expected completion.

   
4. In step 2, change the metric to Custom.

5. Set up the custom metric configuration, which consists:

   • Evaluator model: The model that will evaluate the outputs of your model.

   • Custom evaluator name: Enter a unique name to identify your custom evaluation.

   • Minimum and maximum score: Define the scoring range (example: 0 to 100). If the metric uses inverted scoring, you can set the minimum value higher than the maximum (example: 100 to 0).

   • Prompt: Write a clear, detailed prompt that explains the evaluation process. Include specific criteria and instructions for assessing responses.

   
6. Click Create evaluation project and wait for your evaluation process to finish.

Analyze the evaluation results

After the evaluation process is completed, you can analyze the results. Learn more on how to analyze the result.