Evals

What are Evals?

Evals, short for evaluations, let you compare model and prompt configurations for a Flow using the same test inputs. Review output, success rate, latency, cost, and token usage for each configuration before you change the Flow.

Run an ad hoc Eval from the Flow editor. Choose Realtime for inline inputs or Batch for Records. The Evals page has two tabs: Evals lists saved eval suites, and Runs lists execution history. An eval suite stores test cases and graders for repeatable checks.

Why use Evals?

Use Evals to compare configurations against the requirements for your Flow:

  • Compare models: Run the same Flow with different model choices.
  • Test prompt variations: Change prompt wording and compare the outputs.
  • Tune step settings: Compare temperature, max tokens, reasoning, response format, and tool settings.
  • Check production changes: Measure a modified Flow before you deploy it.
  • Inspect consistency: Review results across representative inputs and edge cases.

How Evals work

Follow this sequence to compare configurations:

  1. Choose a Flow to evaluate.
  2. Choose Realtime or Batch.
  3. Provide test inputs. Enter messages for chat-based Flows or variables for other Flows in Realtime. Select a Record type in Batch.
  4. Define configurations for the model, prompt, and step settings that you want to compare.
  5. Run the Eval. Runtype executes the Flow once for each configuration and test input.
  6. Review the outputs and metrics for each configuration.

Eval components

Ad hoc Evals use test inputs and configurations. Saved eval suites use test cases and graders. This page describes the inputs, overrides, and metrics that support comparisons.

Test dataset

Choose one of the following test input sources:

  • Inline test data: Enter messages or variables for a Realtime Eval without storing them as Records.
  • Record collection: Select a Record type for a Batch Eval. Reuse Records across Evals, and upload a CSV in the Records area to create test data.

Include common cases and edge cases that reflect the inputs that your Flow receives.

Configurations

A configuration is a set of overrides that you compare with other configurations. Configure overrides per prompt step so one Eval can test different settings on different steps.

The available overrides include the following:

  • Model: Compare model choices.
  • Temperature: Change output variation for models that support temperature.
  • Max tokens: Limit response length.
  • Response format: Select JSON, markdown, XML, or HTML output.
  • Reasoning: Configure extended reasoning for supported models.
  • Tools: Add, remove, or change the tools available to the step.

Metrics

Review the following metrics for each configuration:

  • Success rate: Compare completed and failed executions.
  • Latency: Compare duration per step and for the overall execution.
  • Cost: Compare total cost and average cost per Record.
  • Token usage: Compare prompt and completion tokens.
  • Output comparison: Inspect outputs for each Record and configuration.

Use keyword analysis to search for terms across step outputs. Step analysis groups matches by step, model, and Eval.

Choose an execution mode

Choose an execution mode based on your test inputs and how you want to review results:

  • Realtime: Enter messages or variables and watch results stream as the Flow runs.
  • Batch: Select a Record type and queue an Eval that runs in the background. Review the results on the Runs tab of the Evals page.

On Build and Trial plans, Batch Evals use the first 10 Records. Other plans use up to 100 Records.

Evals in practice

The following example is illustrative. It shows how Alex uses a Batch Eval to compare a customer-support Flow.

Set up an Eval

Alex runs customer-support automation for Example Organization. Their team uses a Flow that drafts responses to password resets, billing questions, and feature requests. The team receives feedback about robotic responses and missing context.

Alex’s production configuration uses one model at temperature 0.7. They compare it with two model alternatives and a lower-temperature configuration before changing the Flow.

Alex creates 50 Records from customer questions. The set includes routine questions, frustrated customers, ambiguous requests, and multi-part questions. Alex uploads the Records with a CSV so the team can reuse the test data.

Alex defines four configurations:

  • Production baseline: Keep the production model and temperature setting.
  • Model alternative: Replace the production model with another model.
  • Second model alternative: Replace the production model with a second model.
  • Temperature comparison: Keep the production model and use temperature 0.3.

Alex runs the Eval from the Flow editor. Runtype executes the Flow once for each configuration and Record. Four configurations across 50 Records produce 200 executions.

Review results

Alex reviews the comparison view using the following signals:

  • Output quality: Compare how each configuration handles user frustration and missing context.
  • Latency and cost: Identify configurations that use fewer tokens or complete in less time.
  • Output consistency: Compare how each configuration handles similar inputs.
  • Keyword matches: Search for phrases that match the team’s support style.
  • Edge cases: Find inputs that produce errors or incomplete answers across configurations.

The results show that one prompt omits context for multi-part billing questions. Alex updates the prompt and creates 20 targeted Records for a follow-up Eval.

Choose a configuration

Alex chooses the configuration that meets the team’s quality, latency, and cost requirements. They compare the follow-up results with the production baseline before deploying the change.

When to run Evals

Run Evals in the following situations:

  • Before deployment: Compare a new model or prompt with the production configuration.
  • After a Flow change: Check whether the change affects output quality, latency, cost, or success rate.
  • During model or prompt tuning: Measure the effect of one configuration change at a time.
  • When managing cost or latency: Compare configurations against the requirements for your use case.

To evaluate a Product Surface, use its surface evaluation to compare router settings, orchestration, enabled Capabilities, and model overrides against the same input. To publish a tested Flow for end users, follow Quickstart: Build a social media post generator as a REST API.

Next steps

Continue with these guides: