Performance Monitoring

Overview

Performance Monitoring enhances Automated Evaluation by letting you schedule regular evaluations of your models. Once the interval is set, evaluations run automatically, and you can check the results for timely improvements.

Get started

To use performance monitoring:

1. Navigate to the Performance monitoring page under Evaluation menu.

2. Click Create performance monitoring.

   
3. Configure your evaluation by selecting a model to evaluate and choosing a dataset from the library. If you don’t have one, you can also upload a dataset in a CSV format containing two columns: prompt and expected completion.

   
4. Select the metric, provider, and the evaluator model you want to use for evaluation. Learn more about the evaluators and metrics.

   
5. In the final step, you will need to set and configure the schedule for the evaluation process. You will need to configure:

   1. Recurrence: You can choose the frequency of your evaluation. The available options are:

      1. Daily at 12:00 AM: Your evaluation will be performed on a daily basis at midnight.

      2. Weekly on Sunday at 12:00 AM: Your evaluation will be performed weekly on Sunday at midnight.

      3. Monthly on day at 12:00 AM: Your evaluation will be performed monthly on the first day of the month at midnight.

      4. Custom: You can set and configure your own evaluation frequency.\

         
   2. Monitor performance drift: You can get notified for LLM performance drift over time. Datasaur will notify you via email when any generated completion deviates beyond a specified threshold during scheduled evaluations, indicating potential performance deterioration.

   3. Run immediately: Evaluate your model right away after creating the project, regardless of the recurrence settings.

   
6. Click Create evaluation project, and your performance monitoring project will be created.

   

Evaluation process

Inside the project, you can click Run now to manually start the evaluation process.

Once the evaluation process has started, you will need to wait until it is completed. You'll receive an email once it's finished, or you can refresh the page to see the latest update.

Analyze the evaluation result

After the evaluation process is completed, you can analyze the results.

Summary

On the summary section, you can see the cost and the processing time of the evaluation process. You can also see the average evaluator score and the performance result.

Result and score

In the results section, you can see the completions generated by your model, along with their scores for the selected metric, reasons behind the scores, and overall performance.

Evaluation details

To view the evaluation details of a completion, click the More icon (three dots) at the far right of the row, then select View details.