Managing eval suites

An eval suite is a named collection of test cases and graders attached to one Flow or Agent. Graders score each test case. Unlike ad hoc comparisons in the Flow editor, a suite persists. Rerun it after changes. Manage its definition in your repository for a continuous integration (CI) check.

Use the Evals page

Open Evals in the sidebar to view your suites and run history. The page has two tabs:

• Evals: lists your saved suites, with one row per suite.
• Runs: lists suite runs and ad hoc comparisons from the Flow editor. Use Eval runs or Ad-hoc comparisons in the origin filter, and use the status filter to narrow the run state. Suite-linked runs show a score in the Score column.

View suites

The Evals tab lists your saved suites. Each suite row includes the following fields:

The suite name can include the following badges:

• Default: marks the default suite for the target. A Flow or Agent can have at most one default suite.
• Managed in code: marks a suite whose definition, including its name, target, and graders, is managed in your repository and converged on deploy.

Click a suite row to open its detail page.

View a suite

The suite detail page includes the following sections:

• Definition: shows the description, target, most recent score, and ordered grader list. After you review AI-grader outcomes, Judge trust shows the number of agreements out of total reviews, such as “Agrees with you 7 of 9 times”. Soft graders show a Soft badge. A soft miss is reported but fails the case only in strict mode. For a Managed in code suite, the definition is read-only. Edit it in your repository. For other suites, click Edit to change the name and description, and edit the grader list on the page.
• Test cases: lists enabled and disabled cases. You can edit test cases even when the suite definition is managed in code. Click Add case to create a case with a name, input, optional expected answer, and notes. Click a case row to edit it, or use its row menu to enable, disable, or delete it. Edit complex inputs, including multi-turn messages and expected facts, as JSON. A case captured with Add to eval in Working with Logs shows Replayable when all recorded tool outputs were captured in full. It shows Partial replay when any output was truncated. The next step is still graded with the captured partial results.
• Review queue: holds generated test cases that need your review. Generated cases stay in the queue until you accept them. Each proposal shows its source, rationale, input, and expected details. Generated proposals also show a diversity category and persona. The source badges include Generated, Coverage gap, From a run, and From feedback. Click Accept to add a proposal without changes. Click Edit to change it before accepting. The original proposal stays available for audit. Click Reject to dismiss it. Click Generate cases to propose up to eight cases by default. The generator combines categories such as happy path, edge case, ambiguous input, adversarial input, long input, and formatting stress. It pairs them with personas such as frustrated customers and non-native speakers. It filters out cases without a stable grading bar. Each generation uses one submission from your daily Eval limit. Each suite can receive at most 50 new proposals in a rolling 24-hour period.
• Coverage: reports which target tools and instruction clauses have a matching case or grader. Click Check coverage to view N of M meters and the uncovered items. Each uncovered item has a generate action that proposes cases for that gap. Those proposals enter the review queue. The instruction inventory is cached and refreshes when the target definition changes.
• Runs: lists this suite’s runs, 25 per page, with the most recent runs first. A score-over-time chart appears when at least two scored runs are on the current page. The chart plots completed run scores from that page. Click a run to open its results. For a completed run, use its row menu and click Pin as baseline. The pinned run shows a Baseline badge.

Feedback from conversations can add proposals automatically for a Flow target that already has a suite. A thumbs-down, an NPS rating of six or lower, or a CSAT rating of two or lower creates a regression proposal forked before the rated reply. If you leave a comment, it pre-fills the expected answer. A thumbs-up or an NPS rating of nine or higher creates a golden proposal that uses the endorsed reply as its expected answer. The system proposes each conversation moment only once. It uses the Flow’s default suite, or its only suite when no default exists.

Use the header controls to run the suite. Click Run to run it normally. To apply strict grading, open the run dropdown and click Run in strict mode. Use the overflow menu for Make default for target, Clear baseline, and Delete eval when those actions apply.

Edit graders

To edit graders, open a suite that is not managed in code. The grader editor is on the Definition card. It offers the following options:

• Add grader: opens a palette with Check the output, Check the run, and AI grader options. Output options include Output contains text, Matches the expected answer, and Valid JSON. Run options include Called a tool, Completed without errors, and Cost under a limit.
• Edit and Remove: update or delete a grader from its row. Open a grader to use the Warn instead of fail toggle. A soft grader is reported but does not fail a case unless you use strict mode.
• Split into focused checks: proposes separate checks from an AI grader’s criteria. Review the proposed checks, edit their wording, clear checks that you do not want, and accept the remaining checks. Each accepted check replaces the original with its own AI grader. If the criteria contains one obligation, the editor reports that it is already focused. The SDK method for this operation is Runtype.evals.decomposeCriteria.

Suites managed in code keep their definition, including graders and replay settings, read-only in the dashboard. Change those fields in your repository. Test cases remain editable in the dashboard.

If a suite contains grader types that the dashboard does not recognize, the editor shows their count and preserves them unchanged. Edit these graders with the SDK or API.

View run results

Open a suite-linked run from the suite page or the Runs tab to view its results. The results page shows the following information:

• The headline score counts a case as passed only when every gate grader passes. If the run has soft graders, turn on Strict grading to count soft misses as failures.
• On a large screen, select a case from the case list to view its grader outcomes beside its trace. The page selects the first failed case by default. On smaller screens, cases appear as cards with a collapsible Trace section.
• Each grader outcome includes a Gate or Soft badge. AI-grader outcomes can include reasoning. A soft miss on a non-strict run appears as a warning, not a failure.
• The Insufficient evidence badge means that the available run data does not support a verdict. It does not fail the case. Evidence chips identify the cited step or tool call and can take you to it in the trace.
• For an AI-grader outcome, click 👍 or 👎 to record whether the verdict matches your judgment. A disagreement suggests that you tune the criteria. Your reviews appear in the suite’s Judge trust stat. Click the active thumb again to clear the review.
• When a case has an expected answer, the graded output appears next to it.
• For a checkpoint case captured with Add to eval, next-step mode shows a Next step panel. It compares the recorded action with the current run. It labels the result Next action changed, Same tool, different input, or Same next action.
• The trace lists the steps that ran and each tool call with its inputs and outputs.
• Click Raw results to open the per-record execution view with step details and costs.

Suite-linked runs keep persisted grader scores. Ad hoc comparisons and older runs do not, so their rows open the per-record results sheet.

Run a suite

On the suite page, click Run. From the Evals tab, you can also open a suite’s row menu and click Run. The suite runs every enabled case against its target and scores the output with its graders.

The dashboard uses one of the following execution paths:

• 50 or fewer enabled cases: the suite runs inline, and the scored result appears after completion.
• More than 50 enabled cases: the dashboard queues a durable run and grades it when it completes. Check the Runs tab for the result.

Both paths record the run and its per-case scores in the same run history. The Latest score column and the Runs tab show the same result. Each run counts once against your daily Eval limit.

Delete a suite

On the Evals tab, open a suite’s row menu and click Delete. On the suite page, open the overflow menu and click Delete eval. Deleting a suite removes the suite and its test cases. Past runs keep their persisted scores, so your run history on the Runs tab remains available.

Create a suite

The dashboard does not offer a standalone create-suite form. To create a Flow suite from the dashboard, save a completed Flow run as a test case. In Debugging flows, click Save as test case. The dashboard creates the Flow’s default suite if needed and names it <FLOW_NAME> eval, where FLOW_NAME is the Flow name.

For deliberate suite creation, use one of the following options:

• As code: define the suite in a *.eval.ts file with defineEval, then converge it on deploy with ensure. Use this path for a repository-managed definition and continuous integration (CI) checks.
• With the SDK or API: create and edit suites by ID with client.evals.suites.* in the SDK or with the /v1/eval/suites REST endpoints.

Next steps

Continue with these guides:

• What are Evals?: learn the concepts behind suites, cases, and graders
• Manage evals as code: define suites in your repository and run them as a CI check
• Running an eval: compare models and prompts from the Flow editor
• Interpreting eval results: analyze your results and choose your next change