> For clean Markdown of any page, append .md to the page URL.
> For a complete documentation index, see https://docs.runtype.com/llms.txt.
> For AI client integration (Claude Code, Cursor, etc.), connect to the MCP server at https://docs.runtype.com/_mcp/server.

# Managing eval suites

An eval suite is a named collection of test cases and graders attached to one Flow or Agent. Graders score each test case. A gate grader fails the case when it misses. A soft grader only warns when it misses, unless you run the suite in strict mode. Unlike ad hoc comparisons in the **Flow editor**, a suite is saved, so you can rerun it after each change. You can also manage its definition in your repository for a continuous integration (CI) check.

## Create a suite

The dashboard does not offer a standalone create-suite form. Instead, it creates a target's default suite the first time that you save a run as a test case:

* **Flow suite**: in the **Test** view of the **Run Flow** sheet, click **Save as test case** on a completed run. For details, see [Debugging flows](/user-guide/flows/debugging-flows). The dashboard creates the Flow's default suite if needed and names it `FLOW_NAME eval`, where `FLOW_NAME` is the Flow name.
* **Agent suite**: open an Agent execution in **Logs** and click **Add to eval**. If the Agent has no default suite, select **Create default eval for AGENT_NAME** in the dialog, where `AGENT_NAME` is the Agent name. The dashboard creates the suite when you save and names it `AGENT_NAME eval`. For the capture steps, see [Capture an agent execution as an eval case](/user-guide/logs/working-with-logs#capture-an-agent-execution-as-an-eval-case).

To create a suite without a run, use one of the following options:

* **As code**: define the suite in a `*.eval.ts` file with `defineEval`, then apply it each time you deploy with `Runtype.evals.ensure`. Use this option for a repository-managed definition and CI checks.
* **With the SDK or API**: create and edit suites by ID with `Runtype.evals.suites.*` in the SDK or with the `/v1/eval/suites` REST endpoints. A create request needs a `name` and exactly one of `flowId` or `agentId`. It can also set `graders`, set `isDefault`, and add up to 500 `cases`.
* **With MCP**: call `create_eval_suite`, then add cases with `add_eval_cases` or `add_eval_case_from_execution`. For the full capture loop, see [Improving your agents from production traces](/developer-guides/guides/improving-agents-from-production).

An API key needs `EVALS:WRITE` to create or edit suites and cases, `EVALS:READ`, `EVALS:WRITE`, `FLOWS:READ`, or `FLOWS:EXECUTE` to read them, and `FLOWS:EXECUTE` or `EVALS:WRITE` to run them. Without the permission, the request returns `403` with `code: "FORBIDDEN"`.

## Browse suites on the Evals page

Open **Evals** in the sidebar to view your suites and run history. The page has two tabs:

* **Evals**: lists your saved suites, with one row per suite.
* **Runs**: lists suite runs and ad hoc comparisons from the **Flow editor**. Use **Eval runs** or **Ad-hoc comparisons** in the origin filter, and use the status filter to narrow the run state. Suite-linked runs show a score in the **Score** column.

Each suite row on the **Evals** tab includes the following columns:

| Column           | Meaning                                                                                                                                                                |
| ---------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| **Eval name**    | The suite name and description, with any badges.                                                                                                                       |
| **Target**       | The Flow or Agent that the suite tests.                                                                                                                                |
| **Cases**        | The number of enabled test cases, such as `3 of 5` when some cases are disabled.                                                                                       |
| **Latest score** | The score from the most recent run, shown as passed cases out of graded cases. A case the platform aborted before the target could answer is excluded from this count. |
| **Last run**     | The time of the most recent run.                                                                                                                                       |

The suite name can include the following badges:

* **Default**: marks the default suite for the target. A Flow or Agent can have at most one default suite.
* **Managed in code**: marks a suite whose definition, including its name, target, and graders, is defined in your repository and updated each time you deploy.

Click a suite row to open its detail page.

## View a suite

The suite detail page includes the following sections:

* **Definition**: shows the description, target, most recent score, and ordered grader list. After you review AI-grader outcomes, **Judge trust** shows the number of agreements out of total reviews, such as "Agrees with you 7 of 9 times". Soft graders show a **Soft** badge. For a **Managed in code** suite, the definition is read-only, so edit it in your repository. For other suites, click **Edit** to change the name and description, and edit the grader list on the page.
* **Replay mode**: appears when the suite has a case captured with **Add to eval**. In **Mode**, **Next step** grades only the Agent's first action, and **Continue** serves the recorded tool results so the run keeps going. In **Continue** mode, **Unmatched tool call** sets what happens when a call has no recording: **Fail** or **Stub**. For details, see [Replay mode](/user-guide/evals/manage-evals-as-code#replay-mode).
* **Test cases**: lists enabled and disabled cases. You can edit test cases even when the suite definition is managed in code. Click **Add case** to create a case with a name, input, optional expected answer, and notes. Click a case row to edit it, or use its row menu to enable, disable, or delete it. Edit complex inputs, including multi-turn messages and expected facts, as JSON. For the case fields, see [Define the suite](/user-guide/evals/manage-evals-as-code#define-the-suite).
* **Review queue**: holds proposed test cases that wait for your review. For details, see [Review proposed cases](#review-proposed-cases).
* **Coverage**: reports which target tools and instruction clauses have a matching case or grader. Click **Check coverage** to view `N of M` meters and the uncovered items. After you change the Flow or Agent, click **Refresh** to check again. Each uncovered item has a generate action that proposes cases for that gap in the review queue.
* **Runs**: lists this suite's runs, 25 per page, with the most recent runs first. A score-over-time chart appears when at least two scored runs are on the current page. Click a run to open its results. For a completed run, open its row menu and click **Pin as baseline**. The pinned run shows a **Baseline** badge.

A case captured with **Add to eval** shows **Replayable** when all recorded tool outputs were captured in full. It shows **Partial replay** when any output was truncated. The next step is still graded with the captured partial results.

Use the header controls to run the suite. Click **Run** to run it normally. To apply strict grading, open the run dropdown and click **Run in strict mode**. Use the overflow menu for **Make default for target**, **Clear baseline**, and **Delete eval** when those actions apply.

### Review proposed cases

Proposed cases stay in the review queue until you accept them. Each proposal shows its source, rationale, input, and expected details. The source badge is **Generated**, **Coverage gap**, **From a run**, or **From feedback**.

To review a proposal, do one of the following:

* To add it without changes, click **Accept**.
* To change it first, click **Edit**. The original proposal stays available for audit.
* To dismiss it, click **Reject**.

To propose new cases, click **Generate cases**. The dashboard requests eight cases. The generator combines categories, such as happy path, edge case, and adversarial input, with personas, such as frustrated customers and non-native speakers. Each generated proposal shows its category and persona. When the suite has AI graders, the generator drops cases that those graders cannot score reliably.

Each generation uses one submission from your daily Eval limit. A suite can receive at most 50 new proposals in a rolling 24-hour period. After that, generation stops until older proposals age out of the window. If the generator returns no usable cases, try again. To request between 1 and 12 cases, or to focus generation with a hint such as "angry customers", call `generate_eval_cases` in MCP or `POST /v1/eval/suites/{id}/generate-cases` with `count` and `hint`.

Conversation feedback can also add proposals for a Flow target that already has a suite. A thumbs-down, an NPS rating of six or lower, or a CSAT rating of two or lower creates a regression proposal forked before the rated reply. If the rating includes a comment, the comment pre-fills the expected answer. A thumbs-up or an NPS rating of nine or higher creates a golden proposal that uses the endorsed reply as its expected answer. Feedback goes to the Flow's default suite, or to its only suite when no default exists, and each conversation moment is proposed only once.

### Edit graders

To edit graders, open a suite that is not managed in code. The grader editor is on the **Definition** card. It offers the following options:

* **Add grader**: opens a palette with **Check the output**, **Check the run**, and **AI grader** options. Output options include **Output contains text**, **Matches the expected answer**, and **Valid JSON**. Run options include **Called a tool**, **Completed without errors**, and **Cost under a limit**.
* **Edit** and **Remove**: update or delete a grader from its row. To make a grader soft, open it and turn on **Warn instead of fail**.
* **Split into focused checks**: proposes separate checks from an AI grader's criteria. Review the proposed checks, edit their wording, clear checks that you do not want, and accept the remaining checks. Each accepted check replaces the original with its own AI grader. If the criteria describe one obligation, the editor reports that the grader is already focused. The SDK method for this operation is `Runtype.evals.decomposeCriteria`.

Suites managed in code keep their definition, including graders and replay mode, read-only in the dashboard. Change those fields in your repository. Test cases remain editable in the dashboard.

A case added through the API, SDK, or MCP can set its own `graders`. These graders replace the suite graders for that case only. To use the suite graders again, set the case's `graders` to `null`. In a `*.eval.ts` file, a case's `expect` graders are added to the suite graders instead.

If a suite contains grader types that the dashboard does not recognize, the editor shows their count and keeps them unchanged. Edit these graders with the SDK or API.

## Run a suite

On the suite page, click **Run**. From the **Evals** tab, you can also open a suite's row menu and click **Run**. The suite runs every enabled case against its target and scores the output with its graders. Each run counts once against your daily Eval limit.

The dashboard runs a suite in one of the following ways:

* **50 or fewer enabled cases**: the suite runs while you wait, and the scored result opens when the run completes.
* **More than 50 enabled cases**: the dashboard starts the run in the background and grades it when it completes. Check the suite's **Runs** list for the result.

Both ways save the run and its per-case scores in the same run history, so the **Latest score** column and the **Runs** tab show the same result.

For an Agent suite, **Run** evaluates the version that the Agent runs when a request names no version. For details, see [What runs when you do not name a version](/user-guide/agents/agent-versioning-and-publishing#what-runs-when-you-do-not-name-a-version). To grade a specific release alias or version, run the suite from the CLI, SDK, API, or MCP with a version selector, as described in [Evaluate a specific Agent version](/user-guide/evals/manage-evals-as-code#evaluate-a-specific-agent-version).

### Run against a different model

The dashboard runs each case with the model that the target is configured to use. To compare models without changing the suite or its target, run the suite from the API, SDK, or MCP with a model override. Every case in that run uses the model that you name. Repeat the run once for each candidate model, then compare the scores.

The following code runs a suite with a different model:

```typescript
import { Runtype } from '@runtypelabs/sdk'

Runtype.configure({ apiKey: process.env.RUNTYPE_API_KEY })

const result = await Runtype.evals.suites.run('YOUR_SUITE_ID', {
  modelOverride: 'claude-sonnet-5',
})
```

Replace `YOUR_SUITE_ID` with the suite ID. In the REST API, send `modelOverride` in the body of `POST /v1/eval/suites/{id}/run`. In MCP, pass `model_override` to `run_eval_suite`.

### If a run does not start

A run returns an error instead of starting in these cases:

* **No enabled cases**: enable at least one case in **Test cases**.
* **No graders**: add at least one grader to the suite.
* **Captured cases in a suite with more than 50 enabled cases**: cases captured with **Add to eval**, and cases that replay recorded tool results in **Continue** mode, run only when the suite has 50 or fewer enabled cases. Disable cases, or move the captured cases to a smaller suite. The error code is `REPLAY_MODE_CASES_UNSUPPORTED_ON_BATCH`.
* **Agent not deployed**: when your organization's live-default setting is on and the Agent's `live` alias names nothing runnable, a run without a version selector returns `404` with `code: "AGENT_NOT_DEPLOYED"`. Deploy a version to `live`.
* **Daily Eval limit reached**: the run returns `402` with `code: "DAILY_EVAL_LIMIT"`. The count resets at midnight UTC.
* **Spending limit reached**: the run returns `402` with `code: "SPEND_LIMIT_EXCEEDED"`. For the causes, see [What can return a 402](/user-guide/troubleshooting-faq/rate-limits-and-usage#what-can-return-a-402).

## View run results

Open a suite-linked run from the suite page or the **Runs** tab to view its results. The results page shows the following information:

* The headline score counts a case as passed only when every gate grader passes. If the run has soft graders, turn on **Strict grading** to count soft misses as failures. A case the platform aborted before your Flow or Agent could respond runs no graders and is left out of the headline score, so it counts as neither a pass nor a failure.
* On a large screen, select a case from the case list to view its grader outcomes beside its trace. The page selects the first failed case by default. On smaller screens, cases appear as cards with a collapsible **Trace** section.
* Each grader outcome includes a **Gate** or **Soft** badge. AI-grader outcomes can include reasoning. A soft miss on a non-strict run appears as a warning, not a failure.
* The **Insufficient evidence** badge means that the available run data does not support a verdict. It does not fail the case. Evidence chips identify the cited step or tool call and can take you to it in the trace.
* To record whether an AI-grader verdict matches your judgment, click the thumbs-up button (**Grader got this right**) or the thumbs-down button (**Grader got this wrong**). A disagreement suggests that you tune the criteria. Your reviews appear in the suite's **Judge trust** stat. To clear a review, click the selected button again.
* When a case has an expected answer, the graded output appears next to it.
* For a case captured with **Add to eval** in a **Next step** suite, a **Next step** panel compares the recorded action with the current run. It labels the result **Next action changed**, **Same tool, different input**, or **Same next action**.
* The trace lists the steps that ran and each tool call with its inputs and outputs.
* Click **Raw results** to open the per-case execution view with step details and costs.
* Click **Record runs** to list the run for each case, with its status, duration, and cost. Select a run to open its timeline.

Suite-linked runs keep saved grader scores. Ad hoc comparisons and runs without saved grader scores open the per-case results sheet instead.

In API and SDK results, a platform AI-grader outcome can include `judgeDiagnostics.attempts` with at most two judge attempts. Each attempt reports its model, duration, response status, and available token usage and finish reason. Use `parseError` to distinguish an empty response, malformed JSON, an invalid verdict contract, or truncated output. The metadata excludes prompts, response text, and provider error messages.

When a custom judge flow cannot produce a verdict, `judgeDiagnostics.flowFailure` reports `deadline_exhausted`, `execution_failed`, or `invalid_response`, with an empty `attempts` array. A valid `insufficient_evidence` verdict does not set this failure marker. Older results, check graders, and successful custom judge flows omit `judgeDiagnostics`.

## Delete a suite

On the **Evals** tab, open a suite's row menu and click **Delete**. On the suite page, open the overflow menu and click **Delete eval**. Deleting a suite removes the suite and its test cases. Past runs keep their saved scores, so your run history on the **Runs** tab remains available.

## Next steps

Continue with these guides:

* [What are Evals?](/user-guide/evals/what-are-evals): learn the concepts behind suites, cases, and graders
* [Designing evals that work](/user-guide/evals/designing-evals-that-work): choose graders and build cases from observed failures
* [Maintaining your eval set](/user-guide/evals/maintaining-your-eval-set): keep a suite useful with baselines, soft graders, and score trends
* [Manage evals as code](/user-guide/evals/manage-evals-as-code): define suites in your repository and run them as a CI check
* [Running an eval](/user-guide/evals/running-an-eval): compare models and prompts from the **Flow editor**
* [Interpreting eval results](/user-guide/evals/interpreting-eval-results): analyze your results and choose your next change