Managing eval suites

An eval suite is a saved, named set of test cases and graders attached to one Flow or Agent. Unlike the ad-hoc comparisons you run from the Flow editor, a suite persists: you can rerun it after every change, watch its score over time, and keep it in your repository as a CI gate.

The Evals page has two tabs

Open Evals in the sidebar to find two tabs:

• Evals lists your saved suites, one row per suite.
• Runs holds the execution history: every suite run plus the ad-hoc comparisons started from the Flow editor. Use the origin filter to show only Eval runs (suite runs) or only Ad-hoc comparisons, and the status filter to narrow by run state. Suite-linked runs show their score in the Score column.

The Evals tab

Each suite row shows:

Two badges can appear next to the name:

• Default — the suite is the default eval for its target. A Flow or Agent has at most one default suite.
• Managed in code — the suite’s definition (name, target, graders) lives in your repository and is converged on deploy. See Manage evals as code.

Click a suite row to open its detail page.

The suite page

Each suite has a detail page with everything about that eval in one place:

• Definition — the description, the target Flow or Agent (linked), the latest score, and the ordered grader list. Once you have reviewed AI-grader verdicts on run results (👍/👎), a Judge trust stat appears here as a plain fraction (“Agrees with you 7 of 9 times”) so you know how far to trust the automated verdicts; it nudges you toward tuning the criteria when disagreements pile up. Soft graders carry a Soft badge: their misses are reported but only fail a case in strict mode. For a Managed in code suite the definition is read-only here (your repository is the source of truth and reconverges it on deploy); for other suites, Edit changes the name and description, and the grader list is fully editable on the page (see below).
• Test cases — the suite’s cases, freely editable even when the definition is managed in code. Add a case with Add case (name, input, optional expected answer and notes), click a row to edit it, or use the row menu to enable, disable, or delete it. Complex inputs (multi-turn messages, expected facts) edit as JSON. A case captured from a run (via Add to eval in Logs) also shows a Replayable badge when every recorded tool output was captured in full, or Partial replay when some output was truncated at capture (the case still grades the next step, but with partial tool results).
• Review queue — machine-proposed test cases waiting for your review. Nothing generated enters the suite without your approval: each proposal card shows where it came from (a provenance badge such as Generated, Coverage gap, From a run, or From feedback, plus the diversity category and persona it was generated for), a one-line rationale for why the case matters, and expandable input/expected details. Accept adds it to the suite as-is, Edit lets you adjust the case before accepting (the machine original is kept for audit), and Reject dismisses it. The card’s Generate cases button proposes up to about 8 new cases from the target’s definition, spread across a diversity matrix (happy path, edge case, ambiguous input, adversarial, long input, formatting stress, crossed with user personas such as a frustrated customer or a non-native speaker) and filtered so only cases with a clear pass bar reach the queue. Generation counts one submission against your daily Eval limit, and each suite accepts at most 50 new proposals per rolling day. Real end-user feedback also feeds this queue automatically: when a conversation on the suite’s target Flow gets a thumbs-down, an NPS of 6 or below, or a CSAT of 2 or below, the conversation is proposed as a regression case forked just before the poorly rated reply (a left comment pre-fills the expected answer); a thumbs-up or an NPS of 9 or above proposes it as a golden case whose endorsed reply becomes the expectation. Each conversation moment is proposed once, and only when the Flow already has a suite to land in (its default suite, or its only one).
• Coverage — how well the suite exercises its target. Check coverage reports which of the target’s tools and instruction clauses (“should / should not” obligations in its prompt) have a matching case or grader, as “N of M” meters with a per-item list. Each uncovered item has a Generate button that proposes cases targeted at exactly that gap; they land in the review queue like any other proposal. The instruction analysis is cached and refreshes automatically when the target’s definition changes.
• Runs — this suite’s run history, paginated 25 at a time (most recent first), with a score-over-time chart once two or more scored runs exist on the current page. The chart plots each completed run’s suite score so you can see whether prompt and model changes are moving the eval in the right direction. Click a run to open its results page. Use a completed run’s row menu to Pin as baseline; the pinned run is your comparison reference and carries a Baseline badge.

The header’s Run button runs the suite (its dropdown offers Run in strict mode, where soft misses fail their case), and the overflow menu can make the suite the default for its target, clear the baseline pin, or delete the suite.

Editing graders

For suites that are not managed in code, the grader list on the Definition card is editable in place:

• Add grader opens a palette organized for skimming: output checks first (“Output contains text”, “Matches the expected answer”, “Valid JSON”), then run-behavior checks (“Called a tool”, “Completed without errors”, “Cost under a limit”), then the AI grader with its built-in presets as one-click starters or a write-your-own criteria option.
• Each grader row has Edit and Remove actions, and every grader has a Warn instead of fail toggle that makes it soft (misses are reported but only fail a case in strict mode).
• On an AI grader, Split into focused checks sends the criteria to the platform and proposes focused sub-checks, each independently judged pass or fail. Review the proposal (edit the wording, untick what you don’t want), then accept to replace the single grader with one AI grader per accepted sub-check. If the criteria already states a single obligation, the suite tells you it is already focused. The SDK equivalent is decomposeCriteria.
• Graders created with a newer SDK that this dashboard version does not recognize are listed by count and preserved unchanged on save; edit those with the SDK or API.

Suites managed in code keep the whole definition (graders included) read-only here; change them in your repository.

Run results

Click any suite-linked run (on the suite page or the Runs tab) to open its results page:

• The headline score counts a case as passed when every gate grader passed. A Strict grading toggle appears when the run has soft graders; turning it on recomputes the verdicts with soft misses failing their case.
• On a large screen the page splits into a case list and a detail view: pick a case on the left (the first failed case is selected for you) to see its grader outcomes beside the run’s trace, so a judge’s reasoning and the evidence it cites are on screen together. On smaller screens the cases stack as cards with a collapsible Trace section.
• Every grader outcome shows a Gate or Soft badge and the grader’s reasoning (AI graders explain their verdict). A soft miss on a passing case renders as a warning, not a failure.
• An AI-judge outcome can also carry an Insufficient evidence badge when the judge did not have enough information to decide (this never fails the case), plus evidence chips that highlight the specific step or tool call the verdict cites in the trace.
• Every AI-grader outcome has 👍/👎 buttons: did the grader get this one right? A disagree suggests tuning the criteria so the grader matches your judgment. Your reviews accumulate into the suite page’s Judge trust display (“Agrees with you N of M times”), which quietly flags when graders disagree with you often. Clicking the active thumb again clears the review.
• The graded output appears next to the case’s expected answer when one is set.
• For a checkpoint case (one captured from a run via Add to eval) run in next-step mode, a Next step panel appears above the grader outcomes, comparing the case’s recorded action against what this run did — flagged as Next action changed, Same tool, different input, or Same next action.
• The trace shows the steps that ran and every tool call with its inputs and outputs.
• Raw results opens the per-record execution view with full step detail and costs.

Ad-hoc comparisons and older runs have no persisted grader scores; their rows keep opening the per-record results sheet instead.

Run a suite

Click Run on the suite page, or open a suite’s row menu on the Evals tab and click Run. Every enabled case executes against the suite’s target and is scored by the suite’s graders:

• 50 or fewer enabled cases — the suite runs immediately and the scored result appears as soon as it finishes.
• More than 50 enabled cases — the suite is queued as a durable run and graded when it completes. Watch the Runs tab for the result.

Both paths record the run and its per-case scores in the same run history, so the Latest score column and the Runs tab stay in sync. Each run counts once against your daily eval limit.

Delete a suite

Open a suite’s row menu (or the suite page’s overflow menu) and click Delete. Deleting a suite removes the suite and its test cases, but past runs keep their persisted scores — your run history on the Runs tab survives.

Create a suite

The dashboard doesn’t have a general-purpose create-suite flow, but saving a completed Test run as a test case (see Debugging flows) automatically creates the Flow’s suite the first time you use it, named “<Flow name> eval”. For deliberate suite creation, you have two options:

• As code: define the suite in a *.eval.ts file with defineEval and converge it on deploy with ensure. This is the recommended path — see Manage evals as code.
• Imperatively: create and edit suites by id with client.evals.suites.* in the SDK or the /v1/eval/suites REST endpoints. See Manage suites imperatively.

Next steps

• What are Evals? — the concepts behind suites, cases, and graders
• Manage evals as code — define suites in your repo and run them as a CI gate
• Running an eval — ad-hoc model and prompt comparisons from the Flow editor
• Interpreting eval results — analyze and act on your results