Running an eval

Use evals to compare model and step configuration variations against test data. Choose the configuration that fits your quality, cost, and duration requirements for your Flow. This page covers Flow evals.

For a conceptual overview, read What are Evals?.

Before you begin

Prepare the Flow that you want to evaluate:

The Flow needs at least one prompt step. If it has none, the Eval tab shows “This flow has no configurable AI steps.”
The eval runs the Flow as it is in the editor, including unsaved changes.
If you use Batch mode, create or import Records for a Record type.

Start an eval

To run an eval from the Flow editor, follow these steps:

Open the Flow that you want to evaluate.
Click Run.
Select the Eval tab.
Select Realtime or Batch.
Set up one or more configurations.
Click Run Eval. If you set up more than one configuration, the button shows the configuration count instead, such as Compare 3 Configs.

Choose an execution mode

Choose one of two execution modes:

Realtime

Use Realtime to run one test input immediately and review streaming results. Enter variables for the test input. If the Flow has agent mode steps, also enter messages.

To set the Record name and type that the Flow receives as its input Record, expand Record Settings.

Batch

Use Batch to run a Flow against Records of one selected type. The dashboard queues the eval, so you can leave the page while it runs.

Select a Record Type. The run button stays unavailable until you select one. To compare this run with other runs, enter a Group ID (optional).

Batch mode runs the Flow once per Record for each configuration. Build and Trial plans use up to 10 Records of the type. Other plans use up to 100 Records.

After you submit, the sheet closes and the run appears on the Runs tab of the Evals page. If the selected type has no Records, you see “No records found for type”. Create or import Records first.

Set up configurations

Configurations are the variants that you compare. Each configuration can override settings on prompt steps in your Flow. Each configuration receives a letter badge, such as A, B, or C.

Name each configuration with a clear label, such as Baseline, Cost-focused, or Reasoning-focused.

Settings you can override

You can override the following settings on a prompt step:

Model: Select a model for the step, such as claude-sonnet-5, gemini-3.8-flash, or claude-opus-5.
Temperature: Control how much variation the output can contain.
Max tokens: Limit the response length.
Response format: Leave Use model default on, or turn it off and choose Markdown, JSON, HTML, or XML.
Reasoning: Set reasoning options for supported models.
Tools: Add or remove tools for the step. If you change the model, Runtype removes built-in tools that the new model does not support and shows how many it removed. Check the tools after you change the model.

Example configuration

The following table compares three configurations for one prompt step in Batch mode:

Configuration	Model	Temperature
Baseline	`claude-sonnet-5`	`0.7`
Cost-focused	`gemini-3.8-flash`	`0.7`
Reasoning-focused	`claude-opus-5`	`0.3`

This example changes the model and temperature for one prompt step. In Batch mode, the Flow runs once per Record for each configuration.

Review your results

Review Realtime results in the Flow editor and Batch results on the Evals page.

Realtime results

After a Realtime eval completes, review the comparison table. It shows the output, model, duration, and cost for each configuration and step.

Realtime results are not saved. They stay in the sheet until you click Run Again or close the sheet. To keep results and compare them later on the Evals page, use Batch mode.

Batch results

To review Batch results, follow these steps:

Open the Evals page.
Select the Runs tab. Runs with the same group ID appear together.
To compare grouped runs, click Compare configs.

The Evals tab on the same page lists saved eval suites.

The comparison view contains the following information:

Summary tab: Cards show the Highest success rate, Cheapest (lowest total cost), and Fastest (shortest average duration) configuration, plus the number of evals in the group.
Metrics table: Sort by Success rate, Avg duration, Total cost, Avg cost/record, Output tokens, or Steps. Suite-linked runs also include a Score column with the grader score.
Detail tab: Filter step results by Record or Step Type, compare outputs across configurations, and click Export CSV to download the results.

To count keyword occurrences in step outputs across a group, use the POST /v1/eval/analyze-steps endpoint or the analyze_eval_steps MCP tool.

Run a comparison from the API

To compare configurations without the dashboard, send one POST /v1/eval/submit request with an evalConfigs array. Each entry names a configuration and overrides prompt steps by step ID. The following request runs two configurations against two Records in one group:

curl https://api.runtype.com/v1/eval/submit \
  -H "Authorization: Bearer YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "flowId": "YOUR_FLOW_ID",
    "recordIds": ["YOUR_RECORD_ID_1", "YOUR_RECORD_ID_2"],
    "evalGroupId": "YOUR_GROUP_ID",
    "evalConfigs": [
      {
        "evalName": "Baseline",
        "overrides": { "steps": { "YOUR_STEP_ID": { "model": "claude-sonnet-5" } } }
      },
      {
        "evalName": "Cost-focused",
        "overrides": { "steps": { "YOUR_STEP_ID": { "model": "gemini-3.8-flash" } } }
      }
    ]
  }'

Replace the following:

YOUR_API_KEY: your Runtype API key.
YOUR_FLOW_ID: the ID of the Flow to evaluate.
YOUR_RECORD_ID_1, YOUR_RECORD_ID_2: the IDs of the Records to run.
YOUR_GROUP_ID: a group ID that you choose, so the runs appear together.
YOUR_STEP_ID: the ID of the prompt step to override.

To read the comparison from the CLI, run the following command:

runtype eval compare <group-id>

The Runtype MCP server offers the same actions through the submit_eval and compare_eval tools. To define suites in your repository, see Manage evals as code.

Plan your comparisons

Use these practices to keep comparisons consistent:

Start with Realtime: Run a Realtime eval with one input before a Batch eval.
Include a baseline: Add your production configuration so you can measure changes against it.
Use representative data: Include common cases and edge cases in your Batch Records.
Name configurations clearly: Use labels that distinguish the model and settings in each configuration.
Change one variable at a time: Change either the model or the temperature before you compare results.

For more guidance on test data and graders, see Designing evals that work.

Eval limits

Your plan sets a daily eval limit. The Evals page shows the number of evals that you used and the plan limit when a limit applies. The daily limit applies to each submission, regardless of how many configurations or Records it includes.

The daily count resets at midnight UTC. When you reach the limit, the eval does not start and you see “Daily eval limit reached (N/day). Resets at midnight UTC.” The API returns HTTP 402 with the code DAILY_EVAL_LIMIT. If your organization has a spend limit, an eval can also be blocked with the code SPEND_LIMIT_EXCEEDED.

Next steps

Continue with these guides:

Designing evals that work: choose test data and graders.
Managing eval suites: manage saved suites and graders.
Interpreting eval results: analyze comparisons and choose a configuration.
Manage evals as code: define suites in your repository and run them in CI.
Creating and managing records: prepare data for Batch evals.
Creating and editing flows: build and update the Flow that you evaluate.
Run evals from the API, MCP, and CLI: stream comparisons, submit batch evals, and pull suites without the dashboard.