Skip to navigation
Evals

Interpreting eval results

Use eval results to compare models, prompts, and configurations, and to choose the one that fits your quality, cost, and duration targets.

Before you begin

Run an eval with two or more configurations in the same group. For more information, see Running an eval.

Open a comparison

To open a configuration comparison, follow these steps:

  1. On the Evals page, click Runs.
  2. Find the group of eval runs that you want to compare.
  3. Click Compare configs. The comparison view opens on the Summary tab.

Compare configs appears only on grouped runs, which are configurations submitted together as one comparison.

To see a single run instead, click its row. A suite-linked run opens its run results page, which shows grader outcomes for each case. For more information, see View run results. A run without a suite opens its results in a side panel.

Read the summary

The Summary tab starts with four cards. The following table describes each card:

CardWhat it shows
Highest success rateThe configuration with the highest success rate. When configurations tie, the card shows the tie and how many configurations share it.
CheapestThe configuration with the lowest total cost, among configurations with a success rate above 0%.
FastestThe configuration with the shortest average duration per Record, among configurations with a success rate above 0%.
EvalsThe number of configurations in the group, and the group ID.

Cheapest compares total cost, so a configuration that ran fewer Records can win it. Compare Avg cost/record when Record counts differ. If no configuration has a success rate above 0%, Cheapest and Fastest show No successful evals.

Aggregate metrics

The Aggregate Metrics Comparison table shows one row for each configuration. A trophy marks each configuration that leads on one of the cards. To sort the table, click a column header. Each click cycles through ascending, descending, and unsorted.

The following table describes the columns:

ColumnWhat it showsUse it to
Success rateThe percentage of recorded steps that finish with output.Find configurations with failed steps before you look closer.
Avg durationThe average total step duration per Record.Compare execution time.
Total costThe total estimated cost of the run.Compare overall spending on the same Records.
Avg cost/recordThe average estimated cost per Record.Compare configurations that ran different numbers of Records.
Output tokensThe total completion tokens, when the run records token usage.Compare output length.
StepsSuccessful steps compared with total steps.Count failed steps.

Success rate measures completion, not correctness. A step that returns a wrong answer still counts as a success. To judge quality, compare outputs on the Detail tab.

The comparison view doesn’t show grader scores, because suite runs don’t join a comparison group. To compare graded quality across models, run the suite once for each model and compare the scores on each run’s results page. For more information, see Run against a different model.

Baseline comparison

The Baseline comparison table compares every configuration with the first one, which is labeled (base). Each cell shows the value and its change from the base. Success rate changes are in percentage points. Other changes are in percent. A change shows N/A when the base value is zero and the other value isn’t. The Success rate row appears only when a configuration ran more than one Record.

The table adds the following rows that the aggregate table doesn’t show:

  • Input tokens: the total prompt tokens, when the run records token usage.
  • Cost / 1K runs: the estimated cost of running the configuration 1,000 times, based on its average cost per Record.

Use Cost / 1K runs to estimate spend at production volume.

Inspect step results

To compare outputs Record by Record, follow these steps:

  1. Click the Detail tab. It shows the model, duration, cost, and output of each configuration for each Record and step.
  2. Optional: To narrow the view, select a value in Record or Step Type.
  3. To see a step’s input configuration and Record data, expand the step.
  4. To download the results, click Export CSV.

The CSV file contains only the rows that match the active filters. Each row has the Record name and ID, the step name and type, and the step configuration and Record metadata as JSON. Each configuration adds duration in milliseconds, cost, model, and output columns.

Choose a configuration

Use the following cases to decide which configuration to keep:

  • One configuration leads. If one configuration has the highest success rate and meets your duration and cost targets, review its outputs on the Detail tab. If the outputs meet your quality target, use it.
  • A lower-cost configuration is close. Compare its outputs with the leader’s on the Records that matter most. Choose the higher-cost configuration only when the quality difference matters for your use case.
  • Results vary by Record. If a configuration succeeds on some Records and fails on others, use the Detail tab to find the steps and inputs that fail. Update the prompt or configuration, then run the eval again.
  • Configurations perform alike. If only the prompt differs, improve the prompt before you switch models. If outputs are inconsistent, test a lower temperature with the same model and prompt, especially for classification or extraction.

Act on results

After you choose a configuration, apply it and verify the change:

  1. Update your Flow with the selected model, prompt, and settings.
  2. Run the eval again to confirm that the configuration meets your target.
  3. After you deploy, review production logs to compare real-world behavior.

Read results through the API

You can read the same comparison data with the Runtype API or the Runtype MCP server. The group ID appears on the Evals card on the Summary tab.

The following request returns the aggregate metrics for each configuration in a group:

curl -X POST https://api.runtype.com/v1/eval/compare \
  -H "Authorization: Bearer YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"evalGroupId": "YOUR_EVAL_GROUP_ID"}'

Replace the following:

  • YOUR_API_KEY: your Runtype API key
  • YOUR_EVAL_GROUP_ID: the ID of the eval group

For the full request and response schemas, see Compare evals.

To search step outputs for keywords, send POST /v1/eval/analyze-steps with evalGroupId, stepIds (the IDs of the Flow steps to search), and searchTerms. Set caseSensitive to true to match case. For each configuration and term, the response returns the total occurrences, the number of steps that contain the term, and the occurrence rate. The dashboard doesn’t offer keyword analysis. For more information, see Analyze step results.

The Runtype MCP server provides the following tools for eval results:

  • get_eval_group: list the runs in a group.
  • compare_eval: compare aggregate metrics across a group.
  • compare_eval_record: compare one Record’s step results across a group.
  • get_eval_results: get the aggregate results for one completed run, including pass and fail counts.
  • analyze_eval_steps: count keyword occurrences in step outputs.

Troubleshooting

The following table lists messages that you might see and how to resolve them:

MessageCauseResolution
No step-level data available for this eval groupThe runs in the group have no step results, for example because they haven’t started yet.Wait for the runs to finish, then reload the page.
No group ID providedThe comparison page opened without a group.Open the comparison from Compare configs on the Runs tab.
No grader scores for this runThis message appears on a suite run’s results page. Virtual runs and older runs have no grader scores, or the run is still processing.If the run is still processing, wait for it to finish, then click Refresh.

Next steps

Continue with these guides: