Evals

Designing evals that work

Most eval suites fail in the same way: they test things that never break and miss the things that do. A suite that passes tells you nothing if its cases were invented from a brainstorm instead of drawn from real failures. This guide is about building Evals that actually move quality, not evals that make a green checkmark. It is written for whoever owns the quality bar, which is usually a person with domain judgment, not only an engineer.

Start with error analysis, not a checklist

The instinct is to sit down and brainstorm everything that could go wrong. Resist it. A list of hypothetical failures produces cases that test your imagination, not your system.

Do the opposite. Look at what your Flow or Agent actually did on real inputs first:

  1. Run the thing on a batch of realistic inputs, or pull recent production runs from Logs and the execution trace.
  2. Read the outputs. Not the scores, the actual outputs. Read enough of them that patterns emerge.
  3. Name the failure modes you see and count them. “Invents a refund amount”, “answers in the wrong language”, “ignores the account tier”. Write each one down, and tally how often it happens.

Now you have a ranked list of real problems. Your eval cases come from this list. A grader that does not correspond to a failure you observed, or one you can concretely anticipate, is noise. It costs you a model call and a maintenance burden and tests nothing that was ever going to break.

This is the single highest-leverage habit in eval design. The suite is only as good as the failures it was built from.

Binary pass or fail, never a rating scale

It is tempting to grade quality from 1 to 5. Don’t. Rating scales feel informative and are almost useless: nobody agrees on what a 3 means versus a 4, the numbers drift over time, and an average score hides exactly the failures you care about.

Grade each case as pass or fail against a specific question. Runtype’s graders are built this way. A case passes when every gate grader passes; the suite’s score is passed cases over total cases. That number is honest because each grader answered one yes-or-no question.

If a grader’s criteria contains the word “and”, split it. “Answers the question and uses a professional tone” is two checks pretending to be one. On an AI grader, Split into focused checks does this for you: it proposes one narrow sub-check per obligation, each judged independently. One narrow judge per failure mode beats one broad judge every time, because when a broad judge fails you cannot tell which part failed.

Climb the grader ladder: cheap checks before AI judges

An AI judge is the most expensive and least reliable grader you can use. It costs a model call per case and it can be wrong. Reach for it last.

Runtype’s grader palette is ordered to nudge you down the ladder. Work top to bottom:

  • Deterministic output checks are free and instant. “Output contains text”, “Matches the expected answer”, “Valid JSON”. If the correct answer is a known string, a schema, or a format, a deterministic check settles it with no judgment call.
  • Run-behavior checks assert what the system did, not just what it said. “Called a tool”, “Completed without errors”, “Cost under a limit”. Use these to pin tool-use contracts: it always looked up the order before answering, it never called the refund tool, it finished in one turn.
  • AI graders judge the qualities the checks above cannot express: tone, relevance, whether an answer actually addresses the question. Use one or two of these for the semantic residue, not as your first move.

Domain experts often know a cheap deterministic signal where an engineer reaches for a judge. If your reviewer can say “it’s wrong whenever it omits the ticket number”, that is a contains check, not an AI grader. The reverse also happens: a narrow AI grader that asks “do these two answers mean the same thing” can rescue an over-rigid string comparison that fails on trivial rewording. Pick the cheapest grader that captures the real failure.

Validate an AI judge before you trust it

An AI grader is a model, and a model can be miscalibrated. A judge is not done the moment you write its criteria. Before you trust its scores to gate a decision, check that it agrees with you.

Label a small sample by hand, say 20 to 30 cases, pass or fail, using your own judgment. Then run the AI grader over the same cases and compare. If the judge disagrees with your labels often, its criteria are wrong, not the system under test. Rewrite the criteria and check again.

You don’t need a side spreadsheet for this: on any run’s results page, use the 👍/👎 buttons on each AI-grader outcome to record whether it got the verdict right. Reviews accumulate into a Judge trust fraction on the suite page (“Agrees with you N of M times”), which flags when disagreements pile up so you know it’s time to rewrite the criteria.

This matters because a more capable judge model can move the reported score without improving agreement with humans at all. If the judge becomes more permissive, the suite can look better even though the system under test did not improve. Read the judge’s reasoning on the failures; Runtype shows each AI grader’s verdict and the evidence it cites next to the trace, so you can see whether it is judging the right thing. When a judge lacks the information to decide, it returns Insufficient evidence rather than guessing, which never fails the case; watch for that badge, because a judge that is frequently unsure is a judge whose criteria need sharpening.

A quick smoke test for judge stability: flip the order of the two answers it is comparing, or perturb the input slightly. A judge whose verdict flips with the order is not measuring quality.

Capture production failures as regression cases

The best test cases are the failures you did not see coming. When something goes wrong in production, do not just fix it. Capture it first.

The loop is:

  1. Reproduce and understand the failure from the run’s trace or Logs.
  2. Add the failing input as a new case, with graders or an expected answer that pin the correct behavior. The new case should fail against the current version. Saving a Test run as a case from the Flow debugger is the fast path, and it creates the suite automatically the first time.
  3. Fix the Flow or Agent.
  4. Re-run the suite. The new case flips to pass, and the existing cases prove your fix did not break anything else.

Two rules keep this honest. Write the case to assert what should happen, never paste a wrong-but-current output into the expected answer, or you enshrine the bug. And capture failures of intended behavior, not infrastructure blips: a rate limit or a provider outage is not a regression case. Scrub personal or sensitive data out of any captured input before you save it.

Small suites of real failures beat large synthetic ones

Twenty cases drawn from real failures are worth more than two hundred generated ones. Synthetic cases tend to cluster around the easy, obvious inputs, which are exactly the ones your system already handles. Seed a suite with 5 to 15 realistic cases from actual usage and the edge cases you genuinely worried about, then grow it from production failures over time. Each pass through the capture loop makes the suite more representative of real traffic and harder to regress.

Two hygiene notes on suite size and passing:

  • Do not draw strong conclusions from a handful of cases. A suite of eight cases moving from 6/8 to 7/8 is barely a signal. When your suite is small, say so when you report the result.
  • A green suite is not “production ready”, it is “no known regressions”. Frame a passing suite as a regression harness, not a certificate. If your suite passes 100%, it is too easy; add the harder cases you have been avoiding.

Test every step, not just the ending

For multi-step Flows, an end-to-end pass or fail hides where the trouble started. Upstream failures dominate: a good final answer built on a wrong Record lookup is luck, and a bad final answer might have a perfectly good first step. The compounding math is unforgiving. Eight steps at 95% reliability each land near 66% overall.

So grade the funnel. Add run-behavior graders that assert individual steps ran and ran in the right order (“looked up the customer before drafting”), alongside at least one end-to-end case that checks the whole trajectory. When a step is side-effecting, verify the real state change, not the model’s claim about it. An Agent that says “I saved the record” is not evidence a record was saved; assert against the Record itself. Runtype’s trace view puts each grader’s outcome next to the step it cites, so you can see exactly which stage failed.

For anything used repeatedly, one green run is not the bar. A run that passes once might fail the next time on the same input. If reliability matters, run the suite several times and require it to pass every time, not just on average.

Know when not to build evals

Evals are infrastructure, and infrastructure can become a displacement activity. If a failure is obvious and you can fix it right now, fix it. Capture the regression case afterward so it stays fixed. Do not stand up an elaborate eval harness to avoid the two-line change that solves the problem in front of you.

The role split is worth stating plainly, because it is easy to get backwards. The tooling can pull traces, propose failure taxonomies, scaffold graders, and run experiments. What it cannot do is supply taste. The labels, the criteria language, the tie-breaking on ambiguous cases, and the sign-off on what “good” means are yours. Automated “here’s what’s wrong with your outputs” analysis is a hypothesis generator, never ground truth. The person who owns the quality bar confirms the failure taxonomy before any of it becomes an eval.

Next steps