Evals

Designing evals that work

Design eval suites around observed failures. Make each case test a behavior that matters to you, and define what each grader checks. Use Evals to track whether your changes fix those failures without introducing new ones.

Start with error analysis, not a checklist

Before you write cases, inspect what your Flow or Agent does on realistic inputs. Do not start with a list of hypothetical failures. Start with observed runs.

Use this process:

  1. Run your Flow or Agent on realistic inputs.
  2. Inspect production runs in Logs and their traces.
  3. Read the outputs and traces. Look for recurring patterns, not only scores.
  4. Name and count the failure modes. Write down examples such as “invents a refund amount”, “answers in the wrong language”, or “ignores the account tier”.

Use this list to choose cases and graders. Focus each grader on an observed failure or a concrete risk that you can describe. Otherwise, you cannot tell which decision the grader supports.

Use binary pass or fail checks

Define each case against a specific pass or fail question. A case passes when every gate grader passes. A soft grader miss does not fail the case unless you run the suite in strict mode. The suite score is the number of passed cases divided by the total number of cases.

Keep one obligation in each grader. If a criterion contains the word “and”, split the obligations. For example, “answers the question and uses a professional tone” contains two checks. On an AI grader, Split into focused checks proposes one sub-check for each obligation. Review the proposals before you accept them as separate graders.

Choose the least expensive grader that fits

An AI grader adds a model call for each case and can disagree with your judgment. Start with deterministic checks, then run-behavior checks, and then AI graders.

Use this order:

  • Deterministic output checks: choose Output contains text, Matches the expected answer, or Valid JSON when a known string, schema, or format defines the correct output.
  • Run-behavior checks: choose Called a tool, Completed without errors, or Cost under a limit when you need to assert what the run did.
  • AI graders: use these for qualities that deterministic and run-behavior checks cannot express, such as tone, relevance, or whether the answer addresses the question.

Choose a deterministic check when you can state a precise condition. For example, use a contains check when an output fails whenever it omits a ticket number. Use an AI grader when the requirement allows valid rewording, such as two answers that convey the same meaning.

Validate an AI judge before you trust it

An AI grader can disagree with your judgment. Calibrate it before you use it to gate a decision.

Label 20 to 30 cases by hand as pass or fail. Run the AI grader on the same cases and compare its verdicts with your labels. If the grader disagrees often, rewrite its criteria and repeat the comparison.

On a suite run’s results page, use the 👍/👎 controls on each AI-grader outcome to record whether the verdict matches your judgment. The suite page shows Judge trust as a fraction such as “Agrees with you 7 of 9 times”.

Read the grader’s reasoning and the evidence that it cites in the trace. Insufficient evidence means that the judge does not have enough information to decide. The results page displays this outcome as a warning, and it does not fail the case. A change in the judge model can change the reported score without changing the quality of the system under test. Compare judge agreement before you trust a new score.

Test judge stability by reversing the order of two compared answers or by changing the input slightly. If the verdict changes, review the criteria.

Capture production failures as regression cases

Capture an intended-behavior failure before you fix it. Use this loop:

  1. Reproduce the failure.
  2. Inspect the run’s trace in the Flow debugger.
  3. Add the failing input as a case. Add graders or an expected answer that define the correct behavior.
  4. Run the current version and confirm that the case fails.
  5. Fix the Flow or Agent.
  6. Run the suite again. Confirm that the case passes and that existing cases still pass.

In the Flow debugger, click Save as test case after a test run. The first saved case creates an eval suite for the Flow. For an Agent execution, use Add to eval in Logs to capture the conversation and the next step you want to grade.

Assert intended behavior, not the current behavior. Do not paste a wrong current output into the expected answer. Exclude infrastructure failures such as rate limits and provider outages from regression cases. Remove personal or sensitive data from captured inputs before you save them.

Prefer small suites of real failures

Start with 5 to 15 realistic cases from real usage and edge cases that you care about. Add cases from production failures over time. Prioritize real failures over synthetic cases that cluster around routine inputs.

Apply these rules when you assess suite size and results:

  • Treat a suite with eight cases and a change from 6/8 to 7/8 as limited evidence. Report the suite size with the result.
  • Treat a passing suite as a regression harness, not proof of production readiness.
  • When a suite scores 100%, add cases for known difficult scenarios instead of treating the score as complete coverage.

Test each step and the final result

Check each step of a multi-step Flow as well as its final result. A final result does not show where an earlier failure began. If eight steps each succeed 95% of the time, all eight succeed about 66% of the time when their outcomes are independent.

Add run-behavior graders that assert individual steps ran and that they ran in the required order. Keep at least one end-to-end case that checks the complete result. When a step changes state, verify the state change itself. A message that claims a Record was saved does not prove that the Record was saved. Check the Record.

The run results page shows grader outcomes beside the trace, and evidence chips identify the step or tool call that an AI grader cites. Use this view to identify the stage that failed.

If you rely on repeated behavior, run the suite multiple times and require each run to pass. Do not use an average score as your only requirement.

Decide when not to build evals

Build a suite when you need to check a behavior repeatedly. If you can fix a failure immediately, fix it first. Then capture a regression case so you can check the fix again.

The platform surfaces traces, proposes cases, splits multi-part criteria, and runs comparisons. You define the failure labels, the criteria, and the tie-breaking rules for ambiguous cases. Review proposed cases and grader results before you add a case or use a grader as a gate.

Next steps

Continue with one of these guides: