> For clean Markdown of any page, append .md to the page URL.
> For a complete documentation index, see https://docs.runtype.com/llms.txt.
> For AI client integration (Claude Code, Cursor, etc.), connect to the MCP server at https://docs.runtype.com/_mcp/server.

# Interpreting eval results

Use eval results to compare models, prompts, and configurations, and to choose the
one that fits your quality, cost, and duration targets.

## Before you begin

Run an eval with two or more configurations in the same group. For more information,
see [Running an eval](/user-guide/evals/running-an-eval).

## Open a comparison

To open a configuration comparison, follow these steps:

1. On the **Evals** page, click **Runs**.
2. Find the group of eval runs that you want to compare.
3. Click **Compare configs**. The comparison view opens on the **Summary** tab.

**Compare configs** appears only on grouped runs, which are configurations submitted
together as one comparison.

To see a single run instead, click its row. A suite-linked run opens its run results
page, which shows grader outcomes for each case. For more information, see
[View run results](/user-guide/evals/managing-eval-suites#view-run-results). A run
without a suite opens its results in a side panel.

## Read the summary

The **Summary** tab starts with four cards. The following table describes each card:

| Card                     | What it shows                                                                                                                          |
| ------------------------ | -------------------------------------------------------------------------------------------------------------------------------------- |
| **Highest success rate** | The configuration with the highest success rate. When configurations tie, the card shows the tie and how many configurations share it. |
| **Cheapest**             | The configuration with the lowest total cost, among configurations with a success rate above 0%.                                       |
| **Fastest**              | The configuration with the shortest average duration per Record, among configurations with a success rate above 0%.                    |
| **Evals**                | The number of configurations in the group, and the group ID.                                                                           |

**Cheapest** compares total cost, so a configuration that ran fewer Records can win
it. Compare **Avg cost/record** when Record counts differ. If no configuration has a
success rate above 0%, **Cheapest** and **Fastest** show **No successful evals**.

### Aggregate metrics

The **Aggregate Metrics Comparison** table shows one row for each configuration. A
trophy marks each configuration that leads on one of the cards. To sort the table,
click a column header. Each click cycles through ascending, descending, and unsorted.

The following table describes the columns:

| Column              | What it shows                                                  | Use it to                                                     |
| ------------------- | -------------------------------------------------------------- | ------------------------------------------------------------- |
| **Success rate**    | The percentage of recorded steps that finish with output.      | Find configurations with failed steps before you look closer. |
| **Avg duration**    | The average total step duration per Record.                    | Compare execution time.                                       |
| **Total cost**      | The total estimated cost of the run.                           | Compare overall spending on the same Records.                 |
| **Avg cost/record** | The average estimated cost per Record.                         | Compare configurations that ran different numbers of Records. |
| **Output tokens**   | The total completion tokens, when the run records token usage. | Compare output length.                                        |
| **Steps**           | Successful steps compared with total steps.                    | Count failed steps.                                           |

**Success rate** measures completion, not correctness. A step that returns a wrong
answer still counts as a success. To judge quality, compare outputs on the **Detail**
tab.

The comparison view doesn't show grader scores, because suite runs don't join a
comparison group. To compare graded quality across models, run the suite once for each
model and compare the scores on each run's results page. For more information, see
[Run against a different model](/user-guide/evals/managing-eval-suites#run-against-a-different-model).

### Baseline comparison

The **Baseline comparison** table compares every configuration with the first one,
which is labeled **(base)**. Each cell shows the value and its change from the base.
Success rate changes are in percentage points. Other changes are in percent. A change
shows **N/A** when the base value is zero and the other value isn't. The **Success
rate** row appears only when a configuration ran more than one Record.

The table adds the following rows that the aggregate table doesn't show:

* **Input tokens**: the total prompt tokens, when the run records token usage.
* **Cost / 1K runs**: the estimated cost of running the configuration 1,000 times,
  based on its average cost per Record.

Use **Cost / 1K runs** to estimate spend at production volume.

## Inspect step results

To compare outputs Record by Record, follow these steps:

1. Click the **Detail** tab. It shows the model, duration, cost, and output of each
   configuration for each Record and step.
2. Optional: To narrow the view, select a value in **Record** or **Step Type**.
3. To see a step's input configuration and Record data, expand the step.
4. To download the results, click **Export CSV**.

The CSV file contains only the rows that match the active filters. Each row has the
Record name and ID, the step name and type, and the step configuration and Record
metadata as JSON. Each configuration adds duration in milliseconds, cost, model, and
output columns.

## Choose a configuration

Use the following cases to decide which configuration to keep:

* **One configuration leads.** If one configuration has the highest success rate and
  meets your duration and cost targets, review its outputs on the **Detail** tab. If
  the outputs meet your quality target, use it.
* **A lower-cost configuration is close.** Compare its outputs with the leader's on
  the Records that matter most. Choose the higher-cost configuration only when the
  quality difference matters for your use case.
* **Results vary by Record.** If a configuration succeeds on some Records and fails
  on others, use the **Detail** tab to find the steps and inputs that fail. Update
  the prompt or configuration, then run the eval again.
* **Configurations perform alike.** If only the prompt differs, improve the prompt
  before you switch models. If outputs are inconsistent, test a lower temperature
  with the same model and prompt, especially for classification or extraction.

## Act on results

After you choose a configuration, apply it and verify the change:

1. Update your Flow with the selected model, prompt, and settings.
2. Run the eval again to confirm that the configuration meets your target.
3. After you deploy, review production logs to compare real-world behavior.

## Read results through the API

You can read the same comparison data with the Runtype API or the Runtype MCP server.
The group ID appears on the **Evals** card on the **Summary** tab.

The following request returns the aggregate metrics for each configuration in a
group:

```bash
curl -X POST https://api.runtype.com/v1/eval/compare \
  -H "Authorization: Bearer YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"evalGroupId": "YOUR_EVAL_GROUP_ID"}'
```

Replace the following:

* `YOUR_API_KEY`: your Runtype API key
* `YOUR_EVAL_GROUP_ID`: the ID of the eval group

For the full request and response schemas, see
[Compare evals](/api-reference/runtype-api/evals/compare-evals).

To search step outputs for keywords, send `POST /v1/eval/analyze-steps` with
`evalGroupId`, `stepIds` (the IDs of the Flow steps to search), and `searchTerms`.
Set `caseSensitive` to `true` to match case. For each configuration and term, the
response returns the total occurrences, the number of steps that contain the term,
and the occurrence rate. The dashboard doesn't offer keyword analysis. For more
information, see [Analyze step results](/api-reference/runtype-api/evals/analyze-step-results).

The Runtype MCP server provides the following tools for eval results:

* `get_eval_group`: list the runs in a group.
* `compare_eval`: compare aggregate metrics across a group.
* `compare_eval_record`: compare one Record's step results across a group.
* `get_eval_results`: get the aggregate results for one completed run, including
  pass and fail counts.
* `analyze_eval_steps`: count keyword occurrences in step outputs.

## Troubleshooting

The following table lists messages that you might see and how to resolve them:

| Message                                              | Cause                                                                                                                                  | Resolution                                                                     |
| ---------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------ |
| **No step-level data available for this eval group** | The runs in the group have no step results, for example because they haven't started yet.                                              | Wait for the runs to finish, then reload the page.                             |
| **No group ID provided**                             | The comparison page opened without a group.                                                                                            | Open the comparison from **Compare configs** on the **Runs** tab.              |
| **No grader scores for this run**                    | This message appears on a suite run's results page. Virtual runs and older runs have no grader scores, or the run is still processing. | If the run is still processing, wait for it to finish, then click **Refresh**. |

## Next steps

Continue with these guides:

* [Managing eval suites](/user-guide/evals/managing-eval-suites): read grader outcomes, strict grading, and judge reviews.
* [Designing evals that work](/user-guide/evals/designing-evals-that-work): calibrate AI judges so scores reflect quality.
* [Manage evals as code](/user-guide/evals/manage-evals-as-code): run suites and read scores with the SDK, API, or CLI.
* [Running an eval](/user-guide/evals/running-an-eval): run configuration comparisons.
* [What are Evals?](/user-guide/evals/what-are-evals): understand evals, configurations, and metrics.
* [Working with logs](/user-guide/logs/working-with-logs): review production executions after deployment.