> For clean Markdown of any page, append .md to the page URL.
> For a complete documentation index, see https://docs.runtype.com/llms.txt.
> For AI client integration (Claude Code, Cursor, etc.), connect to the MCP server at https://docs.runtype.com/_mcp/server.

# What are Evals?

Evals, short for evaluations, measure how a [Flow](/user-guide/flows/what-are-flows) or
Agent responds to a fixed set of test inputs. Use them to compare model, prompt, and
step settings before you change a Flow, and to check that a Flow or Agent still meets
your requirements after each change.

Runtype offers two kinds of Evals. The following table compares them:

|                        | Ad hoc comparison                              | Eval suite                                              |
| ---------------------- | ---------------------------------------------- | ------------------------------------------------------- |
| **Purpose**            | Compare configurations side by side.           | Rerun the same checks after every change.               |
| **Where you start it** | The **Flow editor**                            | The **Evals** page, the CLI, or CI                      |
| **Targets**            | Flows                                          | Flows, Agents, and external Agents                      |
| **Test inputs**        | Inline messages or variables, or a Record type | Saved test cases                                        |
| **Result**             | Metrics and outputs for each configuration     | A pass or fail verdict for each case, and a suite score |

The **Evals** page has two tabs. **Evals** lists your eval suites. **Runs** lists eval
runs, including suite runs and ad hoc comparisons.

You can also run and manage Evals with the Runtype CLI and SDK (see
[Manage evals as code](/user-guide/evals/manage-evals-as-code)), the
[REST API](/api-reference), and the
[Runtype MCP server](/developer-guides/integrations/runtype-mcp-server). Through the API
or MCP, you can also batch-compare configurations for an Agent. The dashboard runs ad hoc
comparisons for Flows only.

## When to use Evals

Run Evals in the following situations:

* **Before deployment**: Compare a new model or prompt with the production configuration.
* **After a Flow or Agent change**: Check whether the change affects output quality,
  latency, cost, or success rate.
* **During model or prompt tuning**: Measure the effect of one change at a time, such as
  the model, prompt wording, temperature, or tool settings.
* **When managing cost or latency**: Compare configurations against the requirements for
  your use case.
* **After a production failure**: Save the failing run as a test case so that the suite
  catches the same failure again.

## How ad hoc comparisons work

An ad hoc comparison runs the Flow once for each configuration and test input, then shows
the metrics and outputs for each configuration side by side. For the steps, see
[Running an eval](/user-guide/evals/running-an-eval).

### Test inputs

Each comparison uses one of the following test input sources:

* **Inline test data**: Messages for chat-based Flows, or variables for other Flows. A
  **Realtime** comparison uses inline data and does not store it as Records.
* **Records**: A Record type that a **Batch** comparison runs against. You can reuse
  Records across comparisons, and upload a CSV in the Records area to create them.

Include common cases and edge cases that reflect the inputs that your Flow receives.

### Configurations

A configuration is a set of overrides that you compare with other configurations. You set
overrides per prompt step, so one comparison can test different settings on different
steps. The available overrides include the following:

* **Model**: Compare model choices.
* **Temperature**: Change output variation for models that support temperature.
* **Max tokens**: Limit response length.
* **Response format**: Select JSON, Markdown, XML, or HTML output, or keep the model default.
* **Reasoning**: Configure extended reasoning for supported models.
* **Tools**: Add, remove, or change the tools available to the step.

### Metrics

The comparison view shows the following metrics for each configuration:

* **Success rate**: The percentage of recorded steps that finish with output.
* **Latency**: Duration per step and for the whole run.
* **Cost**: Total cost and average cost per Record.
* **Token usage**: Input and output tokens.
* **Output comparison**: The output for each Record and configuration.

For the full list of metrics and how to read them, see
[Interpreting eval results](/user-guide/evals/interpreting-eval-results).

Keyword analysis is available through the API and the `analyze_eval_steps` MCP tool, not
in the dashboard. For each Eval in a group, it reports each term's total occurrences, the
number of steps that contain the term, and its occurrence rate.

### Execution modes

An ad hoc comparison runs in one of the following modes:

* **Realtime**: You enter messages or variables and watch results stream as the Flow runs.
* **Batch**: You select a Record type, and the comparison runs in the background. The
  results appear on the **Runs** tab of the **Evals** page.

On Build and Trial plans, a Batch comparison from the dashboard uses up to 10 Records
of the selected Record type. On other plans, it uses up to 100 Records. Your plan also sets
a daily Eval limit, and each submission counts once, regardless of how many configurations
or Records it includes.

## Eval suites, test cases, and graders

An eval suite is a saved collection of test cases and graders for one target: a Flow, an
Agent, or an external Agent that you
[connect to Runtype](/developer-guides/guides/bring-your-own-agent). Runtype can rerun a
suite against an external Agent only through its A2A endpoint, and it skips cases that
replay recorded tool calls. For an external Agent that only sends telemetry, you can
capture runs as test cases, but you rerun the scenario yourself.

A suite is built from the following parts:

* **Test case**: An input for the target, such as variables, a message, or a
  conversation, with an optional expected answer. A case can come from a run that you
  saved, from a generated proposal that you accepted, or from manual entry.
* **Grader**: A check that decides whether the target's response to a case passes. Graders
  come in two families:
  * **Checks** test the output or the run directly. For example, a check can test whether
    the output contains a phrase, is valid JSON, called a specific tool, or stayed under a
    latency or cost limit.
  * **AI graders**, also called judges, use a model to grade the response against
    criteria that you write in plain language.
* **Gate and soft graders**: A gate grader fails the case when it misses. A soft grader
  reports the miss but fails the case only in a strict run, such as a CLI run with
  `--strict`. In the dashboard, a soft grader
  has **Warn instead of fail** turned on. In code, you mark it with `.soft()`.
* **Score**: The number of passed cases divided by the number of graded cases. Cases that
  the platform stopped before the target answered are not graded.

For suite setup, see [Managing eval suites](/user-guide/evals/managing-eval-suites). For
advice on choosing graders, see
[Designing evals that work](/user-guide/evals/designing-evals-that-work).

## How eval suites grow

A suite is most useful as a regression check that grows from real failures:

1. **Capture a failure.** For an Agent, click **Add to eval** on a failing run in the
   logs, as described in [Working with logs](/user-guide/logs/working-with-logs). For a
   Flow, click **Save as test case** after a test run, as described in
   [Debugging flows](/user-guide/flows/debugging-flows).
2. **Review proposals.** Runtype can propose new cases from the target's definition, from
   coverage gaps, and from conversation feedback. Proposals wait in the **Review queue**
   until you accept or reject them.
3. **Rerun the suite.** After each change to the Flow or Agent, rerun the suite and compare
   the score with the previous run.

To decide which cases and graders to keep over time, see
[Maintaining your eval set](/user-guide/evals/maintaining-your-eval-set).

## Evals in practice

The following example is illustrative. It shows how Alex uses a Batch comparison to
improve a customer-support Flow.

### Set up the comparison

Alex runs customer-support automation for Example Organization. The team uses a Flow that
drafts responses to password resets, billing questions, and feature requests. The team
receives feedback about robotic responses and missing context.

Alex's production configuration uses one model at temperature 0.7. Alex creates 50 Records
from customer questions, including routine questions, frustrated customers, ambiguous
requests, and multi-part questions. Alex uploads the Records with a CSV so the team can
reuse them.

Alex defines four configurations:

* **Production baseline**: Keep the production model and temperature.
* **Model alternative**: Replace the production model with another model.
* **Second model alternative**: Replace the production model with a second model.
* **Temperature comparison**: Keep the production model and use temperature 0.3.

Alex runs the comparison from the **Flow editor**. Runtype runs the Flow once for each
configuration and Record, so four configurations across 50 Records produce 200 runs.

### Review results

Alex reviews the comparison view using the following signals:

* **Output quality**: How each configuration handles user frustration and missing context.
* **Latency and cost**: Which configurations use fewer tokens or finish in less time.
* **Output consistency**: How each configuration handles similar inputs.
* **Edge cases**: Which inputs produce errors or incomplete answers.

The results show that the temperature 0.3 configuration omits context for multi-part
billing questions, and that the first model alternative handles them well at a lower cost.
Alex creates 20 targeted Records for multi-part questions and runs a follow-up comparison.

### Choose a configuration

Alex chooses the configuration that meets the team's quality, latency, and cost
requirements, and compares the follow-up results with the production baseline before
deploying the change. Alex then saves the failing multi-part questions as test cases so
that the Flow's eval suite catches the problem if it returns.

## Next steps

* [Designing evals that work](/user-guide/evals/designing-evals-that-work): Choose graders
  and build suites from real failures.
* [Maintaining your eval set](/user-guide/evals/maintaining-your-eval-set): Decide which
  cases and graders to keep as your Flow changes.
* [Running an eval](/user-guide/evals/running-an-eval): Run Realtime and Batch comparisons
  from the **Flow editor**.
* [Managing eval suites](/user-guide/evals/managing-eval-suites): View suites, edit
  graders, and review proposals.
* [Interpreting eval results](/user-guide/evals/interpreting-eval-results): Read the
  metrics and output comparisons.
* [Manage evals as code](/user-guide/evals/manage-evals-as-code): Keep eval suites in your
  repository and run them as a CI gate.
* [Working with logs](/user-guide/logs/working-with-logs): Find failing runs and add them
  to an eval suite.
* [Bring your own agent](/developer-guides/guides/bring-your-own-agent): Connect an Agent
  that runs outside Runtype so you can evaluate it.