> For clean Markdown of any page, append .md to the page URL.
> For a complete documentation index, see https://docs.runtype.com/llms.txt.
> For AI client integration (Claude Code, Cursor, etc.), connect to the MCP server at https://docs.runtype.com/_mcp/server.

# Running an eval

Use evals to compare model and step configuration variations against test data. Choose the configuration that fits your quality, cost, and duration requirements for your [Flow](/user-guide/flows/what-are-flows). This page covers Flow evals.

For a conceptual overview, read [What are Evals?](/user-guide/evals/what-are-evals).

## Before you begin

Prepare the Flow that you want to evaluate:

* The Flow needs at least one prompt step. If it has none, the **Eval** tab shows "This flow has no configurable AI steps."
* The eval runs the Flow as it is in the editor, including unsaved changes.
* If you use Batch mode, create or import Records for a Record type.

## Start an eval

To run an eval from the Flow editor, follow these steps:

1. Open the Flow that you want to evaluate.
2. Click **Run**.
3. Select the **Eval** tab.
4. Select **Realtime** or **Batch**.
5. Set up one or more configurations.
6. Click **Run Eval**. If you set up more than one configuration, the button shows the configuration count instead, such as **Compare 3 Configs**.

## Choose an execution mode

Choose one of two execution modes:

### Realtime

Use **Realtime** to run one test input immediately and review streaming results. Enter variables for the test input. If the Flow has agent mode steps, also enter messages.

To set the Record name and type that the Flow receives as its input Record, expand **Record Settings**.

### Batch

Use **Batch** to run a Flow against Records of one selected type. The dashboard queues the eval, so you can leave the page while it runs.

Select a **Record Type**. The run button stays unavailable until you select one. To compare this run with other runs, enter a **Group ID (optional)**.

Batch mode runs the Flow once per Record for each configuration. Build and Trial plans use up to 10 Records of the type. Other plans use up to 100 Records.

After you submit, the sheet closes and the run appears on the **Runs** tab of the **Evals** page. If the selected type has no Records, you see "No records found for type". Create or import Records first.

## Set up configurations

Configurations are the variants that you compare. Each configuration can override settings on prompt steps in your Flow. Each configuration receives a letter badge, such as A, B, or C.

Name each configuration with a clear label, such as Baseline, Cost-focused, or Reasoning-focused.

### Settings you can override

You can override the following settings on a prompt step:

* **Model**: Select a model for the step, such as `claude-sonnet-5`, `gemini-3.8-flash`, or `claude-opus-5`.
* **Temperature**: Control how much variation the output can contain.
* **Max tokens**: Limit the response length.
* **Response format**: Leave **Use model default** on, or turn it off and choose **Markdown**, **JSON**, **HTML**, or **XML**.
* **Reasoning**: Set reasoning options for supported models.
* **Tools**: Add or remove tools for the step. If you change the model, Runtype removes built-in tools that the new model does not support and shows how many it removed. Check the tools after you change the model.

### Example configuration

The following table compares three configurations for one prompt step in Batch mode:

| Configuration     | Model              | Temperature |
| ----------------- | ------------------ | ----------- |
| Baseline          | `claude-sonnet-5`  | `0.7`       |
| Cost-focused      | `gemini-3.8-flash` | `0.7`       |
| Reasoning-focused | `claude-opus-5`    | `0.3`       |

This example changes the model and temperature for one prompt step. In Batch mode, the Flow runs once per Record for each configuration.

## Review your results

Review Realtime results in the Flow editor and Batch results on the **Evals** page.

### Realtime results

After a Realtime eval completes, review the comparison table. It shows the output, model, duration, and cost for each configuration and step.

Realtime results are not saved. They stay in the sheet until you click **Run Again** or close the sheet. To keep results and compare them later on the **Evals** page, use Batch mode.

### Batch results

To review Batch results, follow these steps:

1. Open the **Evals** page.
2. Select the **Runs** tab. Runs with the same group ID appear together.
3. To compare grouped runs, click **Compare configs**.

The **Evals** tab on the same page lists saved eval suites.

The comparison view contains the following information:

* **Summary** tab: Cards show the **Highest success rate**, **Cheapest** (lowest total cost), and **Fastest** (shortest average duration) configuration, plus the number of evals in the group.
* Metrics table: Sort by **Success rate**, **Avg duration**, **Total cost**, **Avg cost/record**, **Output tokens**, or **Steps**. Suite-linked runs also include a **Score** column with the grader score.
* **Detail** tab: Filter step results by Record or **Step Type**, compare outputs across configurations, and click **Export CSV** to download the results.

To count keyword occurrences in step outputs across a group, use the `POST /v1/eval/analyze-steps` endpoint or the `analyze_eval_steps` MCP tool.

## Run a comparison from the API

To compare configurations without the dashboard, send one `POST /v1/eval/submit` request with an `evalConfigs` array. Each entry names a configuration and overrides prompt steps by step ID. The following request runs two configurations against two Records in one group:

```bash
curl https://api.runtype.com/v1/eval/submit \
  -H "Authorization: Bearer YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "flowId": "YOUR_FLOW_ID",
    "recordIds": ["YOUR_RECORD_ID_1", "YOUR_RECORD_ID_2"],
    "evalGroupId": "YOUR_GROUP_ID",
    "evalConfigs": [
      {
        "evalName": "Baseline",
        "overrides": { "steps": { "YOUR_STEP_ID": { "model": "claude-sonnet-5" } } }
      },
      {
        "evalName": "Cost-focused",
        "overrides": { "steps": { "YOUR_STEP_ID": { "model": "gemini-3.8-flash" } } }
      }
    ]
  }'
```

Replace the following:

* `YOUR_API_KEY`: your Runtype API key.
* `YOUR_FLOW_ID`: the ID of the Flow to evaluate.
* `YOUR_RECORD_ID_1`, `YOUR_RECORD_ID_2`: the IDs of the Records to run.
* `YOUR_GROUP_ID`: a group ID that you choose, so the runs appear together.
* `YOUR_STEP_ID`: the ID of the prompt step to override.

To read the comparison from the CLI, run the following command:

```bash
runtype eval compare <group-id>
```

The Runtype MCP server offers the same actions through the `submit_eval` and `compare_eval` tools. To define suites in your repository, see [Manage evals as code](/user-guide/evals/manage-evals-as-code).

## Plan your comparisons

Use these practices to keep comparisons consistent:

* **Start with Realtime**: Run a Realtime eval with one input before a Batch eval.
* **Include a baseline**: Add your production configuration so you can measure changes against it.
* **Use representative data**: Include common cases and edge cases in your Batch Records.
* **Name configurations clearly**: Use labels that distinguish the model and settings in each configuration.
* **Change one variable at a time**: Change either the model or the temperature before you compare results.

For more guidance on test data and graders, see [Designing evals that work](/user-guide/evals/designing-evals-that-work).

## Eval limits

Your plan sets a daily eval limit. The **Evals** page shows the number of evals that you used and the plan limit when a limit applies. The daily limit applies to each submission, regardless of how many configurations or Records it includes.

The daily count resets at midnight UTC. When you reach the limit, the eval does not start and you see "Daily eval limit reached (N/day). Resets at midnight UTC." The API returns HTTP `402` with the code `DAILY_EVAL_LIMIT`. If your organization has a spend limit, an eval can also be blocked with the code `SPEND_LIMIT_EXCEEDED`.

## Next steps

Continue with these guides:

* [Designing evals that work](/user-guide/evals/designing-evals-that-work): choose test data and graders.
* [Managing eval suites](/user-guide/evals/managing-eval-suites): manage saved suites and graders.
* [Interpreting eval results](/user-guide/evals/interpreting-eval-results): analyze comparisons and choose a configuration.
* [Manage evals as code](/user-guide/evals/manage-evals-as-code): define suites in your repository and run them in CI.
* [Creating and managing records](/user-guide/records/creating-and-managing-records): prepare data for Batch evals.
* [Creating and editing flows](/user-guide/flows/creating-and-editing-flows): build and update the Flow that you evaluate.
* [Run evals from the API, MCP, and CLI](/user-guide/evals/evals-via-api-and-cli): stream comparisons, submit batch evals, and pull suites without the dashboard.