Running an eval

Use Evals to compare model and step configuration variations against test data. Choose the configuration that fits your quality, cost, and duration requirements for your Flow.

For a conceptual overview, read What are Evals?.

Before you begin

Prepare the Flow that you want to evaluate. If you use Batch mode, create or import Records for a Record type.

Start an eval

To run an Eval from the Flow editor, follow these steps:

1. Open the Flow that you want to evaluate.
2. Click Run.
3. Select the Eval tab.
4. Select Realtime or Batch.
5. Set up one or more configurations.
6. Click Run Eval. If you set up more than one configuration, the button is labeled with the configuration count instead, such as Compare 3 Configs.

Choose an execution mode

Choose one of two execution modes:

Realtime

Use Realtime to run one test input immediately and review streaming results. For a Flow with agent mode steps, enter messages. For other Flows, enter variables.

Review the comparison as each configuration completes.

Batch

Use Batch to run a Flow against Records of one selected type. The dashboard queues the Eval, so you can leave the page while it runs.

Select a Record Type. To compare this run with other runs, enter a Group ID (optional).

Batch mode runs the Flow once per Record for each configuration. Build and Trial plans use the first 10 Records. Other plans use up to 100 Records.

Set up configurations

Configurations are the variants that you compare. Each configuration can override settings on prompt steps in your Flow. Each configuration receives a letter badge, such as A, B, or C.

Name each configuration with a clear label, such as Baseline, Cost-focused, or Reasoning-focused.

Settings you can override

You can override the following settings on a prompt step:

• Model: Select a model for the step, such as gemini-3.7-flash, gpt-5.6-terra, or claude-sonnet-5.
• Temperature: Control how much variation the output can contain.
• Max tokens: Limit the response length.
• Response format: Choose JSON, markdown, XML, or HTML.
• Reasoning: Set reasoning options for supported models.
• Tools: Add or remove tools for the step.

Example configuration

Use the following example to compare three configurations for one prompt step in Batch mode:

This example changes the model and temperature for one prompt step. In Batch mode, the Flow runs once per Record for each configuration.

Review your results

Review Realtime results in the Flow editor and Batch results on the Evals page.

Realtime results

After a Realtime Eval completes, review the comparison table. It shows the output, model, duration, and cost for each configuration and step.

Batch results

On the Evals page, select the Runs tab. Runs with the same group ID appear together. The Evals tab lists saved eval suites.

To compare grouped runs, click Compare configs. The comparison view contains the following information:

• Summary cards: Show the highest success rate, lowest total cost, and shortest average duration for the group.
• Metrics table: Sort by Success Rate, Avg Duration, Total Cost, Avg Cost/Record, Output Tokens, or Steps. Suite-linked runs also include a Score column with the grader score.
• Detail tab: Filter step results by Record or Step Type, review outputs across configurations, and export the results as CSV.
• Keyword analysis: Search step outputs for terms across an eval group.

Plan your comparisons

Use these practices to keep comparisons consistent:

• Start with Realtime: Run a Realtime Eval with one input before a Batch Eval.
• Include a baseline: Add your production configuration so you can measure changes against it.
• Use representative data: Include common cases and edge cases in your Batch Records.
• Name configurations clearly: Use labels that distinguish the model and settings in each configuration.
• Change one variable at a time: Change either the model or the temperature before you compare results.

Eval limits

Your plan sets a daily Eval limit. The Evals page shows the number of Evals that you used and the plan limit when a limit applies. The daily limit applies to each submission, regardless of how many configurations or Records it includes.

Next steps

Continue with these guides:

• Managing eval suites: manage saved suites and graders.
• Interpreting eval results: analyze comparisons and choose a configuration.
• Manage evals as code: define suites in your repository and run them in CI.
• Creating and managing records: prepare data for Batch Evals.
• Creating and editing flows: build and update the Flow that you evaluate.