Skip to navigation
Evals

Designing evals that work

Design eval suites around observed failures. Make each case test a behavior that matters to you, and define what each grader checks. Use Evals to track whether your changes fix those failures without introducing new ones.

Start with error analysis, not a checklist

Before you write cases, inspect what your Flow or Agent does on realistic inputs. Don’t start with a list of hypothetical failures. Start with observed runs.

Use this process:

  1. Run your Flow or Agent on realistic inputs.
  2. In Logs, read the outputs and traces of those runs and of recent production runs. Look for recurring patterns, not only scores.
  3. Name and count the failure modes. Write down examples such as “invents a refund amount”, “answers in the wrong language”, or “ignores the account tier”.

Use this list to choose cases and graders. If a grader doesn’t map to an observed failure or a concrete risk, you can’t tell which decision its score supports.

Use binary pass or fail checks

Define each case against a specific pass or fail question. Each grader has a severity. A gate grader miss fails the case. A soft grader miss is reported but doesn’t fail the case. To make a grader soft, turn on Warn instead of fail when you edit it. To count soft misses as failures, open the run drop-down menu on the suite page and click Run in strict mode, or pass --strict to runtype eval run.

The suite score is the number of passed cases divided by the number of graded cases. Cases that the platform stopped before your Flow or Agent answered aren’t counted. For details, see Managing eval suites.

Keep one obligation in each grader. If a criterion contains the word “and”, split the obligations. For example, “answers the question and uses a professional tone” contains two checks. On an AI grader, Split into focused checks proposes one sub-check for each obligation. Review the proposals before you accept them as separate graders.

Choose the least expensive grader that fits

An AI grader adds a model call for each case and can disagree with your judgment. Start with deterministic checks, then run-behavior checks, and then AI graders.

Use this order:

  • Deterministic output checks: choose Output contains text, Matches the expected answer, or Valid JSON when a known string, schema, or format defines the correct output.
  • Run-behavior checks: choose Called a tool, Completed without errors, or Cost under a limit when you need to assert what the run did.
  • AI graders: use these for qualities that deterministic and run-behavior checks can’t express, such as tone, relevance, or whether the answer addresses the question.

Choose a deterministic check when you can state a precise condition. For example, use an Output contains text check when an output fails whenever it omits a ticket number. Use an AI grader when the requirement allows valid rewording, such as two answers that convey the same meaning. To accept reworded answers, turn on Compare against the expected answer so that the AI grader uses the case’s expected answer as a reference.

The Add a grader dialog offers ready-made AI graders that you can start from and then edit. Several of them match common failure modes. For example, Same language catches answers in the wrong language, and Grounded, no made-up facts catches invented details such as a refund amount. Other presets include Answers the question, Matches the expected answer, and Refuses when it should.

Validate an AI grader before you trust it

An AI grader can disagree with your judgment. Calibrate it before you use it to gate a decision.

Label 20 to 30 cases by hand as pass or fail. Run the AI grader on the same cases and compare its verdicts with your labels. If the grader disagrees often, rewrite its criteria and repeat the comparison. In a suite that you manage as code, you can also pin the grader’s model. For details, see AI judge grading.

On a suite run’s results page, use the thumbs-up (Grader got this right) and thumbs-down (Grader got this wrong) buttons on each AI-grader outcome. They record whether the verdict matches your judgment. The suite page shows Judge trust as a fraction such as “Agrees with you 7 of 9 times”.

Read the grader’s reasoning and the evidence that it cites in the trace. Insufficient evidence means that the grader doesn’t have enough information to decide. The results page displays this outcome as a warning, and it doesn’t fail the case. A change in the grader’s model can change the reported score without changing the quality of the system under test. Compare grader agreement before you trust a new score.

To test the grader’s stability, rerun the suite on slightly changed inputs. If the verdict changes, review the criteria.

Capture production failures as regression cases

Capture an intended-behavior failure before you fix it. Use this loop:

  1. Reproduce the failure.
  2. Inspect the run’s trace. For a Flow, use the Test view of the Run Flow sheet. For an Agent, use the run inspector or Logs.
  3. Add the failing input as a case. Add graders or an expected answer that define the correct behavior.
  4. Run the current version and confirm that the case fails.
  5. Fix the Flow or Agent.
  6. Run the suite again. Confirm that the case passes and that existing cases still pass.

For a Flow, click Save as test case after a test run in the Run Flow sheet. The first saved case creates an eval suite for the Flow. For more information, see Debugging flows.

For an Agent run, click Add to evals in the run inspector, or Add to eval in Logs. In the Eval list, choose a suite for the Agent. If no suite targets the Agent, Runtype creates a default suite when you save. Under Fork point, choose the recorded action to grade. The conversation is frozen immediately before that action. If the dialog reports that the case is partially replayable, some tool outputs were too large to capture in full, and the case replays the truncated outputs.

Assert intended behavior, not the current behavior. Don’t paste a wrong current output into the expected answer. Exclude infrastructure failures such as rate limits and provider outages from regression cases. Remove personal or sensitive data from captured inputs before you save them.

Prefer small suites of real failures

Start with 5 to 15 realistic cases from real usage and edge cases that you care about. Add cases from production failures over time. Prioritize real failures over synthetic cases that cluster around routine inputs.

Proposed cases go to the suite’s Review queue, and no proposal is added until you accept it. Prefer proposals marked From a run or From feedback. Use Generate cases sparingly, and review each proposal before you accept it. To find tools and instructions that no case tests, open the suite’s Coverage panel. For details, see Managing eval suites.

Apply these rules when you assess suite size and results:

  • Treat a suite with eight cases and a change from 6/8 to 7/8 as limited evidence. Report the suite size with the result.
  • Treat a passing suite as a regression harness, not proof of production readiness.
  • When a suite scores 100%, add cases for known difficult scenarios instead of treating the score as complete coverage.

Test each step and the final result

Check each step of a multi-step Flow as well as its final result. A final result doesn’t show where an earlier failure began. If eight steps each succeed 95% of the time, all eight succeed about 66% of the time when their outcomes are independent.

Add a Ran a step grader to assert that a specific step ran, and a Step order grader to assert that steps ran in the required order. For tool calls, add a Tool order grader. Keep at least one end-to-end case that checks the complete result.

When a step changes state, verify the state change itself. A message that claims a Record was saved doesn’t prove that the Record was saved. Graders check the run and its output, not stored Records. To verify a saved Record, make the Flow output the saved Record and check it with a JSON field grader.

The run results page shows grader outcomes beside the trace, and evidence chips identify the step or tool call that an AI grader cites. Use this view to identify the stage that failed.

If you rely on repeated behavior, run the suite multiple times and require each run to pass. Don’t use an average score as your only requirement.

Decide when to build a suite

Build a suite when you need to check a behavior repeatedly. If you can fix a failure immediately, fix it first. Then capture a regression case so that you can check the fix again.

Keep your judgment in the loop

The platform surfaces traces, proposes cases, splits multi-part criteria, and runs comparisons. You define the failure labels, the criteria, and the tie-breaking rules for ambiguous cases. Review proposed cases and grader results before you add a case or use a grader as a gate.

Next steps

Continue with one of these guides: