What are Evals? | Runtype Documentation

Evals, short for evaluations, measure how a Flow or Agent responds to a fixed set of test inputs. Use them to compare model, prompt, and step settings before you change a Flow, and to check that a Flow or Agent still meets your requirements after each change.

Runtype offers two kinds of Evals. The following table compares them:

	Ad hoc comparison	Eval suite
Purpose	Compare configurations side by side.	Rerun the same checks after every change.
Where you start it	The Flow editor	The Evals page, the CLI, or CI
Targets	Flows	Flows, Agents, and external Agents
Test inputs	Inline messages or variables, or a Record type	Saved test cases
Result	Metrics and outputs for each configuration	A pass or fail verdict for each case, and a suite score

The Evals page has two tabs. Evals lists your eval suites. Runs lists eval runs, including suite runs and ad hoc comparisons.

You can also run and manage Evals with the Runtype CLI and SDK (see Manage evals as code), the REST API, and the Runtype MCP server. Through the API or MCP, you can also batch-compare configurations for an Agent. The dashboard runs ad hoc comparisons for Flows only.

When to use Evals

Run Evals in the following situations:

Before deployment: Compare a new model or prompt with the production configuration.
After a Flow or Agent change: Check whether the change affects output quality, latency, cost, or success rate.
During model or prompt tuning: Measure the effect of one change at a time, such as the model, prompt wording, temperature, or tool settings.
When managing cost or latency: Compare configurations against the requirements for your use case.
After a production failure: Save the failing run as a test case so that the suite catches the same failure again.

How ad hoc comparisons work

An ad hoc comparison runs the Flow once for each configuration and test input, then shows the metrics and outputs for each configuration side by side. For the steps, see Running an eval.

Test inputs

Each comparison uses one of the following test input sources:

Inline test data: Messages for chat-based Flows, or variables for other Flows. A Realtime comparison uses inline data and does not store it as Records.
Records: A Record type that a Batch comparison runs against. You can reuse Records across comparisons, and upload a CSV in the Records area to create them.

Include common cases and edge cases that reflect the inputs that your Flow receives.

Configurations

A configuration is a set of overrides that you compare with other configurations. You set overrides per prompt step, so one comparison can test different settings on different steps. The available overrides include the following:

Model: Compare model choices.
Temperature: Change output variation for models that support temperature.
Max tokens: Limit response length.
Response format: Select JSON, Markdown, XML, or HTML output, or keep the model default.
Reasoning: Configure extended reasoning for supported models.
Tools: Add, remove, or change the tools available to the step.

Metrics

The comparison view shows the following metrics for each configuration:

Success rate: The percentage of recorded steps that finish with output.
Latency: Duration per step and for the whole run.
Cost: Total cost and average cost per Record.
Token usage: Input and output tokens.
Output comparison: The output for each Record and configuration.

For the full list of metrics and how to read them, see Interpreting eval results.

Keyword analysis is available through the API and the analyze_eval_steps MCP tool, not in the dashboard. For each Eval in a group, it reports each term’s total occurrences, the number of steps that contain the term, and its occurrence rate.

Execution modes

An ad hoc comparison runs in one of the following modes:

Realtime: You enter messages or variables and watch results stream as the Flow runs.
Batch: You select a Record type, and the comparison runs in the background. The results appear on the Runs tab of the Evals page.

On Build and Trial plans, a Batch comparison from the dashboard uses up to 10 Records of the selected Record type. On other plans, it uses up to 100 Records. Your plan also sets a daily Eval limit, and each submission counts once, regardless of how many configurations or Records it includes.

Eval suites, test cases, and graders

An eval suite is a saved collection of test cases and graders for one target: a Flow, an Agent, or an external Agent that you connect to Runtype. Runtype can rerun a suite against an external Agent only through its A2A endpoint, and it skips cases that replay recorded tool calls. For an external Agent that only sends telemetry, you can capture runs as test cases, but you rerun the scenario yourself.

A suite is built from the following parts:

Test case: An input for the target, such as variables, a message, or a conversation, with an optional expected answer. A case can come from a run that you saved, from a generated proposal that you accepted, or from manual entry.
Grader: A check that decides whether the target’s response to a case passes. Graders come in two families:
- Checks test the output or the run directly. For example, a check can test whether the output contains a phrase, is valid JSON, called a specific tool, or stayed under a latency or cost limit.
- AI graders, also called judges, use a model to grade the response against criteria that you write in plain language.
Gate and soft graders: A gate grader fails the case when it misses. A soft grader reports the miss but fails the case only in a strict run, such as a CLI run with --strict. In the dashboard, a soft grader has Warn instead of fail turned on. In code, you mark it with .soft().
Score: The number of passed cases divided by the number of graded cases. Cases that the platform stopped before the target answered are not graded.

For suite setup, see Managing eval suites. For advice on choosing graders, see Designing evals that work.

How eval suites grow

A suite is most useful as a regression check that grows from real failures:

Capture a failure. For an Agent, click Add to eval on a failing run in the logs, as described in Working with logs. For a Flow, click Save as test case after a test run, as described in Debugging flows.
Review proposals. Runtype can propose new cases from the target’s definition, from coverage gaps, and from conversation feedback. Proposals wait in the Review queue until you accept or reject them.
Rerun the suite. After each change to the Flow or Agent, rerun the suite and compare the score with the previous run.

To decide which cases and graders to keep over time, see Maintaining your eval set.

Evals in practice

The following example is illustrative. It shows how Alex uses a Batch comparison to improve a customer-support Flow.

Set up the comparison

Alex runs customer-support automation for Example Organization. The team uses a Flow that drafts responses to password resets, billing questions, and feature requests. The team receives feedback about robotic responses and missing context.

Alex’s production configuration uses one model at temperature 0.7. Alex creates 50 Records from customer questions, including routine questions, frustrated customers, ambiguous requests, and multi-part questions. Alex uploads the Records with a CSV so the team can reuse them.

Alex defines four configurations:

Production baseline: Keep the production model and temperature.
Model alternative: Replace the production model with another model.
Second model alternative: Replace the production model with a second model.
Temperature comparison: Keep the production model and use temperature 0.3.

Alex runs the comparison from the Flow editor. Runtype runs the Flow once for each configuration and Record, so four configurations across 50 Records produce 200 runs.

Review results

Alex reviews the comparison view using the following signals:

Output quality: How each configuration handles user frustration and missing context.
Latency and cost: Which configurations use fewer tokens or finish in less time.
Output consistency: How each configuration handles similar inputs.
Edge cases: Which inputs produce errors or incomplete answers.

The results show that the temperature 0.3 configuration omits context for multi-part billing questions, and that the first model alternative handles them well at a lower cost. Alex creates 20 targeted Records for multi-part questions and runs a follow-up comparison.

Choose a configuration

Alex chooses the configuration that meets the team’s quality, latency, and cost requirements, and compares the follow-up results with the production baseline before deploying the change. Alex then saves the failing multi-part questions as test cases so that the Flow’s eval suite catches the problem if it returns.

Next steps

Designing evals that work: Choose graders and build suites from real failures.
Maintaining your eval set: Decide which cases and graders to keep as your Flow changes.
Running an eval: Run Realtime and Batch comparisons from the Flow editor.
Managing eval suites: View suites, edit graders, and review proposals.
Interpreting eval results: Read the metrics and output comparisons.
Manage evals as code: Keep eval suites in your repository and run them as a CI gate.
Working with logs: Find failing runs and add them to an eval suite.
Bring your own agent: Connect an Agent that runs outside Runtype so you can evaluate it.