Managing eval suites | Runtype Documentation

An eval suite is a named collection of test cases and graders attached to one Flow or Agent. Graders score each test case. A gate grader fails the case when it misses. A soft grader only warns when it misses, unless you run the suite in strict mode. Unlike ad hoc comparisons in the Flow editor, a suite is saved, so you can rerun it after each change. You can also manage its definition in your repository for a continuous integration (CI) check.

Create a suite

The dashboard does not offer a standalone create-suite form. Instead, it creates a target’s default suite the first time that you save a run as a test case:

Flow suite: in the Test view of the Run Flow sheet, click Save as test case on a completed run. For details, see Debugging flows. The dashboard creates the Flow’s default suite if needed and names it FLOW_NAME eval, where FLOW_NAME is the Flow name.
Agent suite: open an Agent execution in Logs and click Add to eval. If the Agent has no default suite, select Create default eval for AGENT_NAME in the dialog, where AGENT_NAME is the Agent name. The dashboard creates the suite when you save and names it AGENT_NAME eval. For the capture steps, see Capture an agent execution as an eval case.

To create a suite without a run, use one of the following options:

As code: define the suite in a *.eval.ts file with defineEval, then apply it each time you deploy with Runtype.evals.ensure. Use this option for a repository-managed definition and CI checks.
With the SDK or API: create and edit suites by ID with Runtype.evals.suites.* in the SDK or with the /v1/eval/suites REST endpoints. A create request needs a name and exactly one of flowId or agentId. It can also set graders, set isDefault, and add up to 500 cases.
With MCP: call create_eval_suite, then add cases with add_eval_cases or add_eval_case_from_execution. For the full capture loop, see Improving your agents from production traces.

An API key needs EVALS:WRITE to create or edit suites and cases, EVALS:READ, EVALS:WRITE, FLOWS:READ, or FLOWS:EXECUTE to read them, and FLOWS:EXECUTE or EVALS:WRITE to run them. Without the permission, the request returns 403 with code: "FORBIDDEN".

Browse suites on the Evals page

Open Evals in the sidebar to view your suites and run history. The page has two tabs:

Evals: lists your saved suites, with one row per suite.
Runs: lists suite runs and ad hoc comparisons from the Flow editor. Use Eval runs or Ad-hoc comparisons in the origin filter, and use the status filter to narrow the run state. Suite-linked runs show a score in the Score column.

Each suite row on the Evals tab includes the following columns:

Column	Meaning
Eval name	The suite name and description, with any badges.
Target	The Flow or Agent that the suite tests.
Cases	The number of enabled test cases, such as `3 of 5` when some cases are disabled.
Latest score	The score from the most recent run, shown as passed cases out of graded cases. A case the platform aborted before the target could answer is excluded from this count.
Last run	The time of the most recent run.

The suite name can include the following badges:

Default: marks the default suite for the target. A Flow or Agent can have at most one default suite.
Managed in code: marks a suite whose definition, including its name, target, and graders, is defined in your repository and updated each time you deploy.

Click a suite row to open its detail page.

View a suite

The suite detail page includes the following sections:

Definition: shows the description, target, most recent score, and ordered grader list. After you review AI-grader outcomes, Judge trust shows the number of agreements out of total reviews, such as “Agrees with you 7 of 9 times”. Soft graders show a Soft badge. For a Managed in code suite, the definition is read-only, so edit it in your repository. For other suites, click Edit to change the name and description, and edit the grader list on the page.
Replay mode: appears when the suite has a case captured with Add to eval. In Mode, Next step grades only the Agent’s first action, and Continue serves the recorded tool results so the run keeps going. In Continue mode, Unmatched tool call sets what happens when a call has no recording: Fail or Stub. For details, see Replay mode.
Test cases: lists enabled and disabled cases. You can edit test cases even when the suite definition is managed in code. Click Add case to create a case with a name, input, optional expected answer, and notes. Click a case row to edit it, or use its row menu to enable, disable, or delete it. Edit complex inputs, including multi-turn messages and expected facts, as JSON. For the case fields, see Define the suite.
Review queue: holds proposed test cases that wait for your review. For details, see Review proposed cases.
Coverage: reports which target tools and instruction clauses have a matching case or grader. Click Check coverage to view N of M meters and the uncovered items. After you change the Flow or Agent, click Refresh to check again. Each uncovered item has a generate action that proposes cases for that gap in the review queue.
Runs: lists this suite’s runs, 25 per page, with the most recent runs first. A score-over-time chart appears when at least two scored runs are on the current page. Click a run to open its results. For a completed run, open its row menu and click Pin as baseline. The pinned run shows a Baseline badge.

A case captured with Add to eval shows Replayable when all recorded tool outputs were captured in full. It shows Partial replay when any output was truncated. The next step is still graded with the captured partial results.

Use the header controls to run the suite. Click Run to run it normally. To apply strict grading, open the run dropdown and click Run in strict mode. Use the overflow menu for Make default for target, Clear baseline, and Delete eval when those actions apply.

Review proposed cases

Proposed cases stay in the review queue until you accept them. Each proposal shows its source, rationale, input, and expected details. The source badge is Generated, Coverage gap, From a run, or From feedback.

To review a proposal, do one of the following:

To add it without changes, click Accept.
To change it first, click Edit. The original proposal stays available for audit.
To dismiss it, click Reject.

To propose new cases, click Generate cases. The dashboard requests eight cases. The generator combines categories, such as happy path, edge case, and adversarial input, with personas, such as frustrated customers and non-native speakers. Each generated proposal shows its category and persona. When the suite has AI graders, the generator drops cases that those graders cannot score reliably.

Each generation uses one submission from your daily Eval limit. A suite can receive at most 50 new proposals in a rolling 24-hour period. After that, generation stops until older proposals age out of the window. If the generator returns no usable cases, try again. To request between 1 and 12 cases, or to focus generation with a hint such as “angry customers”, call generate_eval_cases in MCP or POST /v1/eval/suites/{id}/generate-cases with count and hint.

Conversation feedback can also add proposals for a Flow target that already has a suite. A thumbs-down, an NPS rating of six or lower, or a CSAT rating of two or lower creates a regression proposal forked before the rated reply. If the rating includes a comment, the comment pre-fills the expected answer. A thumbs-up or an NPS rating of nine or higher creates a golden proposal that uses the endorsed reply as its expected answer. Feedback goes to the Flow’s default suite, or to its only suite when no default exists, and each conversation moment is proposed only once.

Edit graders

To edit graders, open a suite that is not managed in code. The grader editor is on the Definition card. It offers the following options:

Add grader: opens a palette with Check the output, Check the run, and AI grader options. Output options include Output contains text, Matches the expected answer, and Valid JSON. Run options include Called a tool, Completed without errors, and Cost under a limit.
Edit and Remove: update or delete a grader from its row. To make a grader soft, open it and turn on Warn instead of fail.
Split into focused checks: proposes separate checks from an AI grader’s criteria. Review the proposed checks, edit their wording, clear checks that you do not want, and accept the remaining checks. Each accepted check replaces the original with its own AI grader. If the criteria describe one obligation, the editor reports that the grader is already focused. The SDK method for this operation is Runtype.evals.decomposeCriteria.

Suites managed in code keep their definition, including graders and replay mode, read-only in the dashboard. Change those fields in your repository. Test cases remain editable in the dashboard.

A case added through the API, SDK, or MCP can set its own graders. These graders replace the suite graders for that case only. To use the suite graders again, set the case’s graders to null. In a *.eval.ts file, a case’s expect graders are added to the suite graders instead.

If a suite contains grader types that the dashboard does not recognize, the editor shows their count and keeps them unchanged. Edit these graders with the SDK or API.

Run a suite

On the suite page, click Run. From the Evals tab, you can also open a suite’s row menu and click Run. The suite runs every enabled case against its target and scores the output with its graders. Each run counts once against your daily Eval limit.

The dashboard runs a suite in one of the following ways:

50 or fewer enabled cases: the suite runs while you wait, and the scored result opens when the run completes.
More than 50 enabled cases: the dashboard starts the run in the background and grades it when it completes. Check the suite’s Runs list for the result.

Both ways save the run and its per-case scores in the same run history, so the Latest score column and the Runs tab show the same result.

For an Agent suite, Run evaluates the version that the Agent runs when a request names no version. For details, see What runs when you do not name a version. To grade a specific release alias or version, run the suite from the CLI, SDK, API, or MCP with a version selector, as described in Evaluate a specific Agent version.

Run against a different model

The dashboard runs each case with the model that the target is configured to use. To compare models without changing the suite or its target, run the suite from the API, SDK, or MCP with a model override. Every case in that run uses the model that you name. Repeat the run once for each candidate model, then compare the scores.

The following code runs a suite with a different model:

import { Runtype } from '@runtypelabs/sdk'
Runtype.configure({ apiKey: process.env.RUNTYPE_API_KEY })
const result = await Runtype.evals.suites.run('YOUR_SUITE_ID', {
  modelOverride: 'claude-sonnet-5',
})

Replace YOUR_SUITE_ID with the suite ID. In the REST API, send modelOverride in the body of POST /v1/eval/suites/{id}/run. In MCP, pass model_override to run_eval_suite.

If a run does not start

A run returns an error instead of starting in these cases:

No enabled cases: enable at least one case in Test cases.
No graders: add at least one grader to the suite.
Captured cases in a suite with more than 50 enabled cases: cases captured with Add to eval, and cases that replay recorded tool results in Continue mode, run only when the suite has 50 or fewer enabled cases. Disable cases, or move the captured cases to a smaller suite. The error code is REPLAY_MODE_CASES_UNSUPPORTED_ON_BATCH.
Agent not deployed: when your organization’s live-default setting is on and the Agent’s live alias names nothing runnable, a run without a version selector returns 404 with code: "AGENT_NOT_DEPLOYED". Deploy a version to live.
Daily Eval limit reached: the run returns 402 with code: "DAILY_EVAL_LIMIT". The count resets at midnight UTC.
Spending limit reached: the run returns 402 with code: "SPEND_LIMIT_EXCEEDED". For the causes, see What can return a 402.

View run results

Open a suite-linked run from the suite page or the Runs tab to view its results. The results page shows the following information:

The headline score counts a case as passed only when every gate grader passes. If the run has soft graders, turn on Strict grading to count soft misses as failures. A case the platform aborted before your Flow or Agent could respond runs no graders and is left out of the headline score, so it counts as neither a pass nor a failure.
On a large screen, select a case from the case list to view its grader outcomes beside its trace. The page selects the first failed case by default. On smaller screens, cases appear as cards with a collapsible Trace section.
Each grader outcome includes a Gate or Soft badge. AI-grader outcomes can include reasoning. A soft miss on a non-strict run appears as a warning, not a failure.
The Insufficient evidence badge means that the available run data does not support a verdict. It does not fail the case. Evidence chips identify the cited step or tool call and can take you to it in the trace.
To record whether an AI-grader verdict matches your judgment, click the thumbs-up button (Grader got this right) or the thumbs-down button (Grader got this wrong). A disagreement suggests that you tune the criteria. Your reviews appear in the suite’s Judge trust stat. To clear a review, click the selected button again.
When a case has an expected answer, the graded output appears next to it.
For a case captured with Add to eval in a Next step suite, a Next step panel compares the recorded action with the current run. It labels the result Next action changed, Same tool, different input, or Same next action.
The trace lists the steps that ran and each tool call with its inputs and outputs.
Click Raw results to open the per-case execution view with step details and costs.
Click Record runs to list the run for each case, with its status, duration, and cost. Select a run to open its timeline.

Suite-linked runs keep saved grader scores. Ad hoc comparisons and runs without saved grader scores open the per-case results sheet instead.

In API and SDK results, a platform AI-grader outcome can include judgeDiagnostics.attempts with at most two judge attempts. Each attempt reports its model, duration, response status, and available token usage and finish reason. Use parseError to distinguish an empty response, malformed JSON, an invalid verdict contract, or truncated output. The metadata excludes prompts, response text, and provider error messages.

When a custom judge flow cannot produce a verdict, judgeDiagnostics.flowFailure reports deadline_exhausted, execution_failed, or invalid_response, with an empty attempts array. A valid insufficient_evidence verdict does not set this failure marker. Older results, check graders, and successful custom judge flows omit judgeDiagnostics.

Delete a suite

On the Evals tab, open a suite’s row menu and click Delete. On the suite page, open the overflow menu and click Delete eval. Deleting a suite removes the suite and its test cases. Past runs keep their saved scores, so your run history on the Runs tab remains available.

Next steps

Continue with these guides:

What are Evals?: learn the concepts behind suites, cases, and graders
Designing evals that work: choose graders and build cases from observed failures
Maintaining your eval set: keep a suite useful with baselines, soft graders, and score trends
Manage evals as code: define suites in your repository and run them as a CI check
Running an eval: compare models and prompts from the Flow editor
Interpreting eval results: analyze your results and choose your next change