> For clean Markdown of any page, append .md to the page URL. > For a complete documentation index, see https://docs.runtype.com/llms.txt. > For AI client integration (Claude Code, Cursor, etc.), connect to the MCP server at https://docs.runtype.com/_mcp/server. # Manage evals as code Keep eval suites in your repository beside the Flow or Agent that they test. Run the suites as a continuous integration (CI) gate so a regression stops in the pipeline. This page covers the eval counterpart to managing Flows as code and managing Agents as code. It uses the `ensure` protocol and adds the `run` command to run and grade suites. ## Before you begin You need the following: * The TypeScript SDK in your repository: `npm install @runtypelabs/sdk`. * The `runtype` CLI for the CI gate: `npm install -g @runtypelabs/cli`. * A Runtype API key with the `EVALS:WRITE` scope. The CLI reads it from the `RUNTYPE_API_KEY` environment variable, or you can run `runtype login`. For more information, see [Managing API keys](/user-guide/settings/managing-api-keys). * A saved Flow or Agent to evaluate. A suite refers to its target by name. ## Define the suite Use `defineEval` to create a local definition. It validates the input shape and returns a canonical definition without calling the API. A suite names a **target**, a set of **cases**, and the **graders** that score each case. The target is a saved Flow or Agent identified by its name. The following definition targets a Flow and combines suite-level and case-level graders: ```typescript import { defineEval, contains, judges } from '@runtypelabs/sdk' export default defineEval({ target: { flow: 'support-triage' }, graders: [contains('ticket')], cases: [ { name: 'billing routes to finance', input: { variables: { message: 'I was double charged' } }, expected: { facts: ['routes the ticket to finance'] }, expect: [contains('finance'), judges.answersQuestion()], }, ], }) ``` Save the definition in a `*.eval.ts` or `*.eval.mts` file, for example `evals/support-triage.eval.ts`. Export the suite as the file's **default export**. The CLI loads these files without a build step. It skips `node_modules`, build output folders such as `dist` and `build`, and every folder whose name starts with a dot, so don't put suites under a folder such as `.github/`. `defineEval` throws an error when the definition breaks one of these rules: * The top level accepts only `name`, `target`, `graders`, `cases`, `virtual`, `recordedToolMode`, `recordedToolUnmatchedPolicy`, and `identityDefaults`. To set a suite `description`, use the dashboard or `suites.update`. * Each case accepts only `name`, `input`, `expected`, and `expect`. A case `input` also accepts `endUser` and `tenant`, the identity the case runs as. * `cases` can't be empty. * Case names must be unique within the suite, because the name identifies the case. * Every case needs at least one grader, from the suite `graders` or its own `expect`. A case's `expected` object holds the reference answer. It accepts `text`, `json`, and `facts` (a list of strings). Graders such as `matchesExpected()` and `judges.matchesExpected()` compare the output with it. The suite's identity combines its `name` and your account scope. If you omit `name`, it defaults to `flow:` or `agent:`. A target can have one unnamed suite. Give suites for the same target distinct names, such as `smoke` and `regression`, to keep them separate. ### Graders Graders run against each case's final output. Deterministic checks run without a model. AI judges call a model. The SDK provides the following grader builders: | Builder | Checks | | ---------------------------------------- | ----------------------------------------------------------------------------------------------------------- | | `contains(value)` / `notContains(value)` | Output contains or doesn't contain a substring. Case-insensitive unless you pass `{ caseSensitive: true }`. | | `regex(pattern, flags?)` | Output matches a regular expression | | `validJson()` / `jsonField(path, opts?)` | Output parses as JSON, or a dot-path field matches | | `matchesExpected()` | Output equals the case's `expected.text` after normalization | | `length({ minChars, maxChars })` | Output length is within the specified bounds | | `latency(maxMs)` / `noError()` | End-to-end latency is within budget, or the case produces output without an error | | `judge(criteria, opts?)` / `judges.*` | A large language model (LLM) grades the run against plain-language criteria | The `judges` object provides ready-made AI judges with built-in criteria: | Preset | Passes when the response | | ------------------------------ | ----------------------------------------------------------------------- | | `judges.answersQuestion()` | Directly addresses what the user asked | | `judges.matchesExpected()` | Conveys the same facts and conclusion as the case's `expected` answer | | `judges.followsInstructions()` | Obeys every instruction in the system prompt | | `judges.grounded()` | Makes only claims that the provided context or expected answer supports | | `judges.rightTone(voice)` | Matches the voice that you describe in `voice` | | `judges.safeToSend()` | Contains nothing embarrassing to show a customer | | `judges.staysOnTask()` | Stays within its job when the input tries to redirect it | | `judges.sameLanguage()` | Uses the same language as the user's message | | `judges.refusesWhenItShould()` | Declines and redirects an out-of-scope or disallowed request | | `judges.noPersonalData()` | Contains no personal data that the user didn't provide | Suite-level `graders` run on every case. The case's `expect` graders are added after the suite-level graders, so each case runs both lists. Put shared checks in `graders`, and put input-specific checks in each case's `expect` list. The following definition applies a shared check and different case-specific checks: ```typescript import { calledTool, contains, defineEval, judges, noError, usedNoTools } from '@runtypelabs/sdk' export default defineEval({ target: { agent: 'support-agent' }, graders: [noError()], cases: [ { name: 'refund request issues a refund', input: { variables: { message: 'I want a refund for order #1002' } }, expect: [calledTool('issue_refund'), judges.answersQuestion()], }, { name: 'greeting stays a plain reply', input: { variables: { message: 'hi there' } }, expect: [usedNoTools(), contains('help')], }, ], }) ``` The refund case uses `noError`, `calledTool('issue_refund')`, and `judges.answersQuestion()`. The greeting case uses `noError`, `usedNoTools`, and `contains('help')`. ### AI judge grading A `judge` grader evaluates the whole run, not only its final output. If the run has a captured execution trace, the judge uses the recorded steps and tool calls, including their inputs and outputs, with the final output. This supports process criteria such as confirming an order number before issuing a refund or avoiding a call to `delete_record`. To grade against a reference answer, set `useExpected: true`. The judge then receives the case's `expected` object with the run: ```typescript import { judge } from '@runtypelabs/sdk' judge('The reply gives the same refund amount as the expected answer.', { useExpected: true }) ``` For a conversation-seeded case, the judge also receives the seeded conversation that the run answers. Write criteria that refer to that conversation without restating it in the expected answer. If deterministic checks also run, the judge receives their pass or fail outcomes as ground truth. Write judge criteria for qualitative aspects that checks can't express, such as tone, faithfulness, and reasoning quality. Use [trace graders](#trace-graders) for exact deterministic sequence assertions. The judge returns one of three verdicts: * **Pass**: The criterion is clearly met. * **Fail**: The criterion is clearly not met. * **Insufficient evidence**: The run doesn't provide enough information to decide. This verdict renders as a warning and doesn't fail the case, including under `--strict`. Each verdict includes the judge's reasoning. Run results in the dashboard include **evidence chips** that link to the steps or tool calls that support the verdict. Runtype grades the run against your criteria with a default judge model. If that model's provider is unavailable, Runtype falls back to a model from another provider. The default judge also uses family-aware routing. If it shares a creator family with the model that produced the output, Runtype selects a comparable judge from a different family. If no usable cross-family judge is available to your account, Runtype keeps the default judge. Set `model` on a `judge` grader to pin a judge model. A pinned model is used exactly as written, regardless of family routing. Judges with `judgeFlowId` also bypass family routing. #### Custom judge flows When the platform judge prompt doesn't cover your criteria, set `judgeFlowId` to run one of your Flows as the judge. For example, use a retrieval step, rubric lookup, or multi-step deliberation in the judge Flow. The following grader uses a Flow ID placeholder: ```typescript import { judge } from '@runtypelabs/sdk' judge('The answer follows the current returns policy.', { judgeFlowId: 'YOUR_FLOW_ID', }) ``` Replace `YOUR_FLOW_ID` with the ID of the Flow that grades the case. Your Flow runs through the normal Flow pipeline and counts like any other run. It follows this contract: * **Inputs**: The Flow receives `criteria`, `caseName`, `expected`, and `transcript` as Flow variables. The `{{transcript}}` value contains the indexed run trajectory and final output. For conversation-seeded cases, it starts with an unindexed `Seeded conversation` section. If the case has check graders, it also contains an unindexed `Deterministic check results` section with each check's ground-truth outcome. * **Output**: The final output must be the platform verdict JSON: `{"reasoning": string, "verdict": "pass" | "fail" | "insufficient_evidence", "evidence": [{"kind": "message" | "tool_call", "index": number}]}`. The grader validates this contract when it runs. If the judge Flow is missing, fails, or returns invalid verdict JSON, the grader returns an insufficient-evidence warning that names the problem. The warning can't fail or silently pass the case. When `judgeFlowId` is set, `model` is ignored. Each Flow step selects its own model. #### Split a large criterion into focused checks Split criteria that combine several obligations into focused judges. For example, separate confirming an order number from avoiding delivery-date promises. Each judge then returns a clear verdict for one obligation. Use `decomposeCriteria` to propose the split: ```typescript import { judge, Runtype } from '@runtypelabs/sdk' const { subChecks } = await Runtype.evals.decomposeCriteria( 'Confirms the order number before issuing a refund, and never promises a delivery date.' ) const graders = subChecks.map((subCheck) => judge(subCheck.criteria)) ``` Review the proposal before you save it. Edit the wording, remove unwanted sub-checks, and save each accepted sub-check as its own `judge` row. The call doesn't save anything. The resulting graders are ordinary AI graders, so family-aware routing and `.soft()` apply. A single returned sub-check means that the criterion is already focused. ### Trace graders Output graders evaluate the final output. **Trace graders** evaluate recorded behavior, including which tools and steps run, their order, completion status, and cost. Trace graders are deterministic and free. They use the captured execution trace. Use them when the assertion concerns behavior instead of text. The SDK provides the following trace grader builders: | Builder | Checks | | ------------------------- | ------------------------------------------------------------------------------------------------------------------------------------ | | `calledTool(name, opts?)` | A tool named `name` was called. Use `input` or `output` for deep equality, `isError` to match errors, or `times` for an exact count. | | `notCalledTool(name)` | No tool named `name` was called. | | `usedNoTools()` | The run made no tool calls. | | `maxToolCalls(max)` | The run made at most `max` tool calls. | | `toolOrder(tools)` | The listed tools were called in the listed relative order. Other calls can interleave. | | `ranStep(name)` | A step with this name or type ran. | | `stepOrder(steps)` | The listed steps ran in the listed relative order. | | `completed()` | The run finished without an error and wasn't left paused. | | `cost(maxUsd)` | The run cost at most `maxUsd` US dollars. | The following definition checks tool usage and run completion: ```typescript import { calledTool, completed, defineEval, maxToolCalls, notCalledTool } from '@runtypelabs/sdk' export default defineEval({ target: { agent: 'support-agent' }, graders: [completed()], cases: [ { name: 'refund request uses the refund tool, not the delete tool', input: { messages: [{ role: 'user', content: 'I want a refund for order 123' }], }, expect: [calledTool('issue_refund'), notCalledTool('delete_order'), maxToolCalls(3)], }, ], }) ``` For `ranStep` and `stepOrder`, a step matches by configured name, Flow step ID, or step type. For example, `ranStep('prompt')` checks that any prompt step ran. `stepOrder(['Draft reply', 'send-email'])` combines a prompt name with a context-step type. The trace records actual execution order, so `stepOrder` also works for interleaved Flows such as `prompt`, then `upsert-record`, then `prompt`. An eval run records every step and tool call, including steps that don't stream to end users. If a run produces no trace, trace graders fail with an explanatory reason instead of passing silently. ### Severity: gate vs soft Each grader is a **gate** by default. A failed gate fails its case. Mark a grader **soft** with `.soft()` to report a miss without failing the case. A soft miss fails only when the run uses `--strict`. Mark latency, cost, or subjective `judge` checks soft when you want to track them without blocking a merge. Keep correctness checks as gates. The following definition uses gate and soft graders: ```typescript import { contains, defineEval, judge, latency } from '@runtypelabs/sdk' export default defineEval({ target: { flow: 'support-triage' }, cases: [ { name: 'billing routes to finance', input: { variables: { message: 'I was double charged' } }, expect: [ contains('finance'), // gate; a miss fails the case latency(3000).soft(), // soft; a miss is reported judge('The reply is empathetic.').soft(), // soft; requires a pass verdict ], }, ], }) ``` Chain `.gate()` or `.soft()` from any grader builder. `.gate()` is the default. A soft miss appears as a warning in the run output. The judge returns pass or fail, so don't use `.atLeast()` in new suites. Any value from 2 to 5 requires a pass verdict, and `.atLeast(1)` makes the grader always pass. ## Converge at deploy time Use `ensure` to make the saved suite match your repository definition. It creates, updates, or leaves the suite unchanged. It doesn't run the suite. Call it from continuous integration and continuous delivery (CI/CD), alongside the rest of your deployment. Pass your eval definition as `suite` to the `ensure` method: ```typescript import suite from './evals/support-triage.eval' import { Runtype } from '@runtypelabs/sdk' Runtype.configure({ apiKey: process.env.RUNTYPE_API_KEY }) const result = await Runtype.evals.ensure(suite) // { result: 'unchanged' | 'created' | 'updated', suiteId, contentHash } ``` `ensure` first sends a probe with the suite name and content hash. If the hash matches, it writes nothing. If the content differs, it upserts the suite and replaces the cases that `ensure` owns. Cases that you add in the dashboard, save from a run, generate, or import stay in the suite. The target Flow or Agent resolves by name, so the same definition converges in every environment. Your API key selects the account that you update. If a suite with the same name was last changed in the dashboard or through the API, `ensure` returns a `409` with the code `external_modification` and changes nothing. Pass `onConflict: 'overwrite'` to take the suite over, or run `runtype eval run --overwrite-managed` from the CLI. Eval suites keep no version history, so an overwrite permanently replaces edited cases that `ensure` owns. After a takeover, `ensure` marks the suite **Managed in code**. If your definition names a different target under the same suite name, `ensure` points the existing suite at the new target. Pass `dryRun: true` to see the planned change without writing it. The plan returns the live `remoteHash`; pass it back as `expectedRemoteHash` to apply only if nobody changed the suite in between. > **Warning** > > Renaming a suite doesn't rename the existing suite. `ensure` creates another suite under the new > name and leaves the old suite in place. It doesn't delete the old suite. Treat the name as the > stable identity. ### Colocate evals with a Flow Attach eval suites to a Flow definition with an inline `evals` array. Converging the Flow also converges its inline evals in the same call. An inline eval without `target` uses the Flow that contains it. The following Flow definition includes an inline eval suite: ```typescript import { Runtype, contains, defineFlow } from '@runtypelabs/sdk' const supportTriage = defineFlow({ name: 'support-triage', steps: [ /* Add the Flow steps here. */ ], evals: [ { cases: [ { name: 'billing routes to finance', input: { variables: { message: 'I was double charged' } }, expect: [contains('finance')], }, ], }, ], }) const result = await Runtype.flows.ensure(supportTriage) // { result, flowId, versionId, contentHash, evals: [{ result, suiteId, contentHash }] } ``` Inline evals don't change the Flow's content hash. The response returns each suite's converge outcome under `evals`. To run a colocated suite, pass its returned `suiteId` to `Runtype.evals.runSuite({ suiteId })`, or review the suite in the dashboard. The `runtype eval run` command discovers standalone `*.eval.ts` files, not suites converged inline with a Flow. ### Handle dashboard edits A suite can change outside your repository, for example when someone edits its graders in the dashboard. Decide which version wins: * **Code wins**: Run `ensure` with `onConflict: 'overwrite'`. An edit made outside your repository makes a plain `ensure` return `409`, so the overwrite writes your repository definition back, including the cases that `ensure` owns. * **Dashboard wins**: Run `pull`, copy the returned definition into your `*.eval.ts` file, review the `git` diff, then commit it or discard it. Use this call to pull the saved definition and its provenance: ```typescript const { definition, contentHash, lastModifiedSource, updatedAt } = await Runtype.evals.pull('smoke') ``` The pulled definition lists each case's full grader set under `expect`. It contains only the cases that `ensure` owns, so cases saved from runs stay out of your repository. A suite that `ensure` never wrote returns its dashboard cases, and the next `ensure` adopts identical cases instead of duplicating them. A name that matches no suite returns `404`. Adding a case doesn't change the content hash, and `ensure` never deletes a case that it doesn't own. Editing or deleting a case that `ensure` owns counts as an edit outside your repository. When your repository definition changes, `ensure` replaces only its own cases with the cases in your definition. ## Run as a CI gate Use the `runtype eval run` CLI command to discover your `*.eval.ts` files, converge each suite, run every case against its target, grade the outputs, and return an exit code. A regression produces a non-zero exit code. The following commands run suites, select strict grading, or scope discovery: ```bash runtype eval run # discover **/*.eval.ts, ensure, run, score runtype eval run support-triage # filter by path prefix, filename, or suite name runtype eval run --strict # soft grader misses fail the build too runtype eval run --virtual # run inline without persisting a suite runtype eval run --junit .runtype/junit.xml runtype eval run --url # send requests to a different API base URL runtype eval run --cwd packages/support-bot # scope *.eval.ts discovery to a subdirectory runtype eval run --alias # evaluate the version a release alias points at runtype eval run --version-id # evaluate one exact version snapshot ``` Replace the following: * ``: the base URL of the Runtype API to run against, such as a self-hosted deployment * ``: a release alias on the target Agent, such as `pr-482` * ``: an Agent version ID from `GET /v1/agent-versions/{agentId}` The command returns the following exit codes: | Exit code | Meaning | | --------- | -------------------------------------------------------------------------------------------------------------------------------------------- | | `0` | Every case passes every gate grader. Soft misses are reported but don't fail the run. | | `1` | A gate grader fails. With `--strict`, any grader, including a soft grader, fails. | | `2` | The suite couldn't run. For example, no `*.eval.ts` file exists, a file fails to load, authentication fails, or the API rejects the request. | To make soft graders fail the build, pass `--strict`. Without it, soft misses appear as warnings and don't change the exit code. The following GitHub Actions steps install your dependencies and the CLI, then run the gate: ```yaml env: RUNTYPE_API_KEY: ${{ secrets.RUNTYPE_API_KEY }} steps: - uses: actions/checkout@v4 - run: npm ci - run: npm install -g @runtypelabs/cli - run: runtype eval run --junit .runtype/junit.xml ``` Your eval files import `@runtypelabs/sdk`, so install your repository's dependencies before the gate runs. The API key selects the account that the gate runs against. To gate a separate environment, store that environment's API key as its own secret. The target Flows and Agents must exist in that account before the gate runs. Converge them in an earlier step with `Runtype.flows.ensure` in a deploy script, or with `runtype agents ensure`. A non-zero exit blocks the merge. The `run` command first converges the suite definition with `ensure`, so the saved suite remains visible in the dashboard. Saved-suite gate runs also save scores by default. Each run receives a `runId`, which the CLI prints and the API returns. The saved outcomes build a run-by-run history that you can read with `GET /v1/eval/runs/{runId}/scores`. Pass `virtual: true` in the API request, or pass `--virtual` to the CLI, to keep the run ephemeral. ### Limits and supported targets The gate runs each suite synchronously. It has these limits: * A suite runs at most 50 cases per run. * Case execution has a 4-minute total budget. Cases that don't run within the budget are reported as failed with an explanatory message, and the rest of the response is still scored. * Keep slow, multi-step suites well under 50 cases. For larger or slower suites, use a batch eval. The gate supports Flow, standard Agent, and external Agent targets, with these differences for external Agents: * **Delegate mode**: A multi-turn case is sent as one message, and tool traces come only from what the external Agent reports. * **Managed mode**: Case messages stay a multi-turn conversation, and Runtype records the tool calls it makes. * **Replay cases**: Cases that replay recorded tool activity don't run against an external Agent in either mode. Each one is reported as a failed case with a message that explains why. The other cases still run. A Claude Managed Agent isn't supported by the gate. Use a batch eval for that target. ### Batch-evaluate a delegated external Agent Use `POST /v1/eval/submit` to run a batch eval against a Delegate-mode external Agent. Supply one baseline eval config and put each record's conversation in `metadata.messages`: ```bash curl https://api.runtype.com/v1/eval/submit \ -H "Authorization: Bearer $RUNTYPE_API_KEY" \ -H "Content-Type: application/json" \ -d '{ "agentId": "YOUR_AGENT_ID", "records": [ { "name": "refund request", "metadata": { "messages": [ { "role": "user", "content": "I need a refund for order 123" } ] } } ], "evalConfigs": [{ "evalConfigId": "baseline", "evalName": "External baseline" }] }' ``` Replace `YOUR_AGENT_ID` with the ID of the external Agent. Runtype can't see inside the remote endpoint, so this batch evaluates only the Agent's saved remote configuration. Omit `overrides` (an empty object is also accepted) and `options.modelOverride`. Model, step, sub-Agent, Claude Managed, and advisor overrides return `EXTERNAL_AGENT_OVERRIDE_NOT_APPLICABLE` instead of running an unchanged baseline. The batch has no hosted-model snapshot because Runtype doesn't control the remote model. Supplying more than one eval config returns `EXTERNAL_AGENT_COMPARISON_NOT_SUPPORTED`. ### Run inline without persisting Set `virtual: true` on a suite, or pass `--virtual`, to run it without converging anything to the dashboard. This mode doesn't save the suite or its results, and it skips `ensure`. Use it for local checks. ## Manage suites imperatively Use `ensure` when your repository owns the suite definition. When you want to inspect suites, edit cases, or manage a suite outside a repository, use the SDK's ID-addressed create, read, update, and delete (CRUD) methods in the `Runtype.evals.suites` namespace. These methods use the `/v1/eval/suites` REST API. The following code creates a suite, adds a case, and runs the suite: ```typescript import { Runtype, contains } from '@runtypelabs/sdk' Runtype.configure({ apiKey: process.env.RUNTYPE_API_KEY }) const suite = await Runtype.evals.suites.create({ name: 'smoke', flowId: 'YOUR_FLOW_ID', graders: [contains('finance')], cases: [ { name: 'billing routes to finance', input: { variables: { message: 'I was double charged' } }, }, ], }) await Runtype.evals.suites.addCases(suite.id, [ { name: 'refund request', input: { variables: { message: 'I want a refund' } } }, ]) const result = await Runtype.evals.suites.run(suite.id, { strict: true }) if ('mode' in result) { console.log(`queued as batch run ${result.runId}`) } else { console.log(`score ${result.score} (${result.passedCases}/${result.totalCases})`) } ``` Replace `YOUR_FLOW_ID` with the ID of the Flow to evaluate. In these methods, a case's `graders` replace the suite `graders` for that case. This differs from `defineEval`, where a case's `expect` list adds to the suite graders. To combine them, list both. Set `graders: null` in `updateCase` to go back to the suite graders. The SDK maps the following methods to REST endpoints: | SDK method | Endpoint | Purpose | | ------------------------------------------------------ | --------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `suites.create(input)` | `POST /v1/eval/suites` | Creates a suite attached to one Flow or Agent, optionally with initial cases. | | `suites.list(params?)` | `GET /v1/eval/suites?flowId=&agentId=` | Lists suites with case counts and each suite's most recent run and score. | | `suites.get(suiteId)` | `GET /v1/eval/suites/{id}` | Gets a suite with its cases and `judgeAgreement`. Its `agree` count is the reviewed AI-grader verdicts that you agreed with, out of `total` reviewed verdicts. It's `null` until you review a verdict with `reviewScore`. | | `suites.update(suiteId, input)` | `PATCH /v1/eval/suites/{id}` | Updates `name`, `description`, `graders`, `isDefault`, or `baselineBatchExecutionId`. | | `suites.delete(suiteId)` | `DELETE /v1/eval/suites/{id}` | Deletes a suite and its cases. Past runs keep their saved scores. | | `suites.run(suiteId, options?)` | `POST /v1/eval/suites/{id}/run` | Runs every enabled case and grades the outputs. Options are `strict`, `mode` (`sync` or `batch`), `agent`, and `modelOverride`. | | `suites.addCases(suiteId, cases)` | `POST /v1/eval/suites/{id}/cases` | Adds one or more test cases. | | `suites.addCaseFromExecution(suiteId, input)` | `POST /v1/eval/suites/{id}/cases/from-execution` | Captures a case from a real Agent run. | | `suites.updateCase(suiteId, caseId, input)` | `PATCH /v1/eval/suites/{id}/cases/{caseId}` | Edits, enables, or disables a case. | | `suites.deleteCase(suiteId, caseId)` | `DELETE /v1/eval/suites/{id}/cases/{caseId}` | Deletes a case. Past runs keep its saved scores. | | `suites.listProposals(suiteId, params?)` | `GET /v1/eval/suites/{id}/proposals` | Lists the review queue. Filter proposals with `status`, such as `'proposed'`. | | `suites.acceptProposal(suiteId, proposalId, options?)` | `POST /v1/eval/suites/{id}/proposals/{proposalId}/accept` | Accepts a proposal as a case, with an optional edited body in `options.case`. | | `suites.rejectProposal(suiteId, proposalId)` | `POST /v1/eval/suites/{id}/proposals/{proposalId}/reject` | Rejects a proposal. The proposal row remains for audit. | | `suites.generateCases(suiteId, input?)` | `POST /v1/eval/suites/{id}/generate-cases` | Generates case proposals from the target definition across a category and persona matrix. | | `suites.getCoverage(suiteId)` | `GET /v1/eval/suites/{id}/coverage` | Reports which target tools and instruction clauses the suite's cases and graders exercise. | The `acceptProposal` method saves a proposal as a case with `origin: 'generated'`. A trace-sourced proposal uses `origin: 'saved_from_run'`. The accepted case keeps a link back to its proposal. If you pass an edited body in `options.case`, the method records the acceptance as `edited_accepted`. The `addCaseFromExecution` method freezes an Agent conversation at a fork point, attaches recorded tool results as editable mocks, and saves the case with `origin: 'saved_from_run'`. It returns `case` and `replayable` fields. A value of `false` for `replayable` means that a tool output was truncated during capture. The capture doesn't run the Agent again. These methods can't change a suite's **target**. To evaluate a different Flow or Agent, create another suite, or change the target in your `defineEval` definition and run `ensure`. ### Replay mode By default, a captured `saved_from_run` case grades only the Agent's next action. The default is `recordedToolMode: 'next_step'`. During replay, the first tool call is recorded as intent in the trace with its name and arguments, but it isn't run. A text reply is graded as the output. Use a captured case with an Agent-target suite. A Flow-target suite reports an actionable failure instead of running the case. Set `recordedToolMode` to `'continue'` in `defineEval` to serve `toolMocks` by tool name and continue after the first action. Graders can then evaluate what the Agent does after a tool result returns. For example, they can evaluate whether the Agent apologizes after a failed refund. For a suite that you manage outside a repository, send `recordedToolMode` in the body of `POST /v1/eval/suites` or `PATCH /v1/eval/suites/{id}`. The TypeScript types for `suites.create` and `suites.update` don't include this field. A replay doesn't call the real tool. It also works with a `toolType: 'local'` tool, which otherwise pauses on its first call while a client resolves it. When a replayed call has no matching mock, `recordedToolUnmatchedPolicy` selects the outcome. Set it to `'fail'` for the default failure, or to `'stub'` for a canned `tool unavailable` success that lets the run continue. You don't need a captured case to use `toolMocks`. Add them to a manual case's `input` without a `checkpoint`: ```typescript import { contains, defineEval } from '@runtypelabs/sdk' export default defineEval({ target: { agent: 'product-builder' }, recordedToolMode: 'continue', cases: [ { name: 'proceeds past a passing product validation', input: { variables: { productName: 'Example Organization Widgets' }, toolMocks: [{ toolName: 'validate_product', output: { valid: true, issues: [] } }], }, expect: [contains('looks good')], }, ], }) ``` In this example, `validate_product` doesn't run. The mock's `output` is served instead. Edit the `output` on a captured or hand-authored case to test an alternative result. For example, set a lookup result to `not_found` or set `isError` to `true` to test recovery. Continue mode requires an Agent target. ### Grade an agent that runs as an end user An Agent with a [tenancy strategy](/developer-guides/guides/end-user-identity#scope-an-agent-to-the-verified-end-user) needs an end user or a tenant on every run. Give each case the identity to run as. A case's `input` accepts the same `endUser` and `tenant` objects that the body of `POST /v1/dispatch` accepts: * `endUser`: `{ id, email?, ... }`. Other keys are passed through. * `tenant`: `{ id, ... }`. Other keys are passed through. The suite runs as your organization, with your API key or dashboard session. Runtype trusts that caller, so it accepts the identity as given and projects it as it projects a dispatch. The case uses the same end-user record and record scope. Memory-enabled agents remain unsupported in evals. Tools receive the identity as `_endUser.*` and `_tenant.*`. An `external` tool that references `{{_identity.token}}` receives a real signed assertion for that end user. To keep the identity inside Runtype, mock the tool by name with `toolMocks` and set `recordedToolMode: 'continue'`. To give every case the same identity, set `identityDefaults` on the suite. A case that sets its own `endUser` or `tenant` uses its value for that scope and the suite default for the other scope. The following `POST /v1/eval/suites` body sets a default end user and overrides it in one case: ```json { "name": "orders-agent", "agentId": "YOUR_AGENT_ID", "identityDefaults": { "endUser": { "id": "eval-user-1", "email": "eval-user-1@example.com" }, "tenant": { "id": "acme" } }, "graders": [{ "kind": "called_tool", "name": "list_orders" }], "cases": [ { "name": "lists open orders", "input": { "messages": [{ "role": "user", "content": "Show my open orders" }] } }, { "name": "user with no orders", "input": { "messages": [{ "role": "user", "content": "Show my open orders" }], "endUser": { "id": "eval-user-empty" } } } ] } ``` Replace `YOUR_AGENT_ID` with the ID of the Agent to evaluate. The second case runs as `eval-user-empty` within the `acme` tenant. To change the defaults later, send `identityDefaults` in `PATCH /v1/eval/suites/{id}`. `GET /v1/eval/suites/{id}` returns it. To set a case's identity, send `endUser` or `tenant` in the case `input` of `POST /v1/eval/suites/{id}/cases` or `PATCH /v1/eval/suites/{id}/cases/{caseId}`. In the Runtype MCP server, pass `identity_defaults` to `create_eval_suite` or `update_eval_suite`. Case inputs use the same `endUser` and `tenant` keys in `add_eval_cases` and `update_eval_case`. The following `add_eval_cases` arguments add a case that runs as one end user: ```json { "suite_id": "YOUR_SUITE_ID", "cases": [ { "name": "lists open orders for a second user", "input": { "messages": [{ "role": "user", "content": "Show my open orders" }], "endUser": { "id": "eval-user-2" } } } ] } ``` Replace `YOUR_SUITE_ID` with the suite ID. In the dashboard, set the case identity under **Run as end user** in the case dialog, and the suite default under **Identity defaults** in the suite settings. In a suite managed as code, set `identityDefaults` in `defineEval`, and set `endUser` or `tenant` in a case `input`. `ensure` updates the suite when the value changes, because `identityDefaults` is part of the content hash: ```ts import { defineEval } from '@runtypelabs/sdk' export default defineEval({ name: 'orders-agent', target: { agentId: 'YOUR_AGENT_ID' }, identityDefaults: { endUser: { id: 'eval-user-1' }, tenant: { id: 'acme' }, }, graders: [{ kind: 'called_tool', name: 'list_orders' }], cases: [ { name: 'user with no orders', input: { messages: [{ role: 'user', content: 'Show my open orders' }], endUser: { id: 'eval-user-empty' }, }, }, ], }) ``` A case for an Agent with a tenancy strategy fails when neither the case nor the suite supplies the identity that the strategy requires. Its `error` names the missing identity and ends with "Add input.endUser (or input.tenant) to the case, or identityDefaults on the suite." You can't pass an identity proof (`identityProof`) to a suite run. A proof is a short-lived credential for an untrusted caller, such as a browser. A suite runs as your organization, so it doesn't need one, and a proof saved on a case would expire. `run_eval_suite` and `POST /v1/eval/suites/{id}/run` don't accept an identity override. Set the identity on the case or the suite. ### Run a suite by ID Suites with up to 50 enabled cases run synchronously and return the same scored shape as `POST /v1/eval/run`. Larger suites return HTTP `202` with `mode: 'batch'` and a `runId`. The API queues the run and grades it when it completes. Poll `GET /v1/eval/runs/{runId}/scores` for the results. Pass `mode` to force either path. Conversation-seeded cases, which include `messages` in `input`, run on both paths. In a batch run, each case keeps its seeded conversation, and the judge receives it as it does on a synchronous run. Replay-mode cases are the exception. Captured cases and cases whose `toolMocks` are served with `recordedToolMode: 'continue'` run only on the synchronous path. A batch run containing one of these cases is rejected with `REPLAY_MODE_CASES_UNSUPPORTED_ON_BATCH`. Both synchronous and batch runs save per-case scores in the same run history. Each run counts once against the daily eval limit. ### Compare models on the same suite To choose a model based on graded results, run the same suite once for each candidate model with `modelOverride`. Every case runs against that model, and the suite and its target stay unchanged: ```typescript const haiku = await Runtype.evals.suites.run('YOUR_SUITE_ID', { modelOverride: 'claude-haiku-4-5' }) const sonnet = await Runtype.evals.suites.run('YOUR_SUITE_ID', { modelOverride: 'claude-sonnet-5' }) ``` Replace `YOUR_SUITE_ID` with the ID of the suite to run. Compare the scores of the two runs. `POST /v1/eval/run` also accepts `modelOverride` in its request body. ### Evaluate a specific Agent version An Agent-target run can name the version it evaluates instead of taking the saved configuration. `POST /v1/eval/suites/{suiteId}/run`, `POST /v1/eval/run`, and `POST /v1/eval/submit` all accept an `agent` selector carrying exactly one of `alias` or `versionId`: ```json { "agent": { "alias": "pr-482" } } ``` The MCP `run_eval_suite` and `submit_eval` tools expose the same choice as `agent_alias` and `agent_version_id`. The CLI's `runtype eval run` takes `--alias` or `--version-id`, and the SDK's `Runtype.evals.suites.run` takes an `agent` selector. The selector belongs to the run, not to the suite. A suite is a fixed case set, and you choose which version to grade it against each time you run it. The run resolves the selector once, before its first case, so moving the alias during the run doesn't split the run across two versions. Selecting an Agent version pins its configuration, not its [Skill bindings](/user-guide/agents/agent-skills). Agent-target suite runs resolve the Agent's current enabled bindings once for the run. Each binding uses its pinned Skill version, or the Skill's current published version when none is pinned. Saved suite runs record the resolved `slug`, `skillId`, and `skillVersionId` values in `evalConfig.skillBindings`. To keep Skill content constant across runs, pin the bindings to Skill versions and keep the bindings unchanged. Every run records what it evaluated, and the run reads return it with the scores: * `agentVersionId`: the version that ran. * `agentTargetResolution`: `alias`, `version`, or `legacy-live-row` for a run that named no selector while the Agent still ran its saved configuration. * `agentTargetAlias`: the alias that the run named. * `caseManifestHash`: a SHA-256 hash of the case set that the run used. * `evaluatorFingerprint`: a SHA-256 hash of the judges and grading configuration. Runtype records this evidence but doesn't enforce it. Runtype never requires a passing run before a version is deployed, and never blocks or reverses a deployment because of a failing run. A run that resolved a single target gives you the exact version ID to activate. A run that names no selector resolves the same way as any other request to that Agent that names no version. For details, see [What runs when you do not name a version](/user-guide/agents/agent-versioning-and-publishing#what-runs-when-you-do-not-name-a-version). A run fails with one of the following errors when Runtype can't resolve the version: | Status | Code | Cause and fix | | ------ | --------------------------------- | ------------------------------------------------------------------------------------------------------------------------- | | `404` | `alias_not_found` | No alias has that name on the Agent. Check the alias name. | | `404` | `alias_archived` | The alias was archived. Name an active alias, or deploy to a new one. | | `404` | `alias_expired` | The preview alias expired 14 days after its last deploy. Deploy to the alias again, then rerun the gate. | | `404` | `version_not_found` | No version has that ID. Check the version ID. | | `404` | `AGENT_NOT_DEPLOYED` | The run named no selector, the live-default setting is on, and the Agent has no live version. Deploy a version to `live`. | | `422` | `selector_unsupported_agent_type` | External and Claude Managed Agents don't accept a selector. Remove it. | | `400` | Not set | The suite targets a Flow, which doesn't accept a selector. Remove it. | When a suite targets an Agent whose `live` alias is archived, Runtype warns you when you save the suite, before the first run. The create, read, and update responses for that suite carry a `warnings` entry with `code: "AGENT_NOT_DEPLOYED"`. Deploy a version to that Agent's `live` alias to clear it. ### Review generated case proposals Generated cases don't enter a suite directly. `generateCases` creates candidates from the target definition across a category and persona matrix. Categories include happy path, edge case, ambiguous, adversarial, long input, and formatting stress. Personas include a frustrated customer and a non-native speaker. The API filters candidates for gradeability. It reports dropped candidates in `droppedCount` and stores the remaining candidates as **proposals**. Each proposal stores its `source`, `sourceRefs` with the assigned category and persona, and a one-sentence `rationale`. Review each proposal with `acceptProposal` or `rejectProposal`. Accept a proposal as written, or pass `options.case` to save an edited body. Accepted proposals link to the created case through `acceptedCaseId`. The `getCoverage` method reports target tools and instruction clauses without an exercising case or grader. Pass one gap to `generateCases`, such as `{ gap: { toolName } }` or `{ gap: { instruction } }`, to create targeted proposals with `source: 'coverage_gap'`. Each generation call reserves one daily-eval quota slot. A suite accepts at most 50 new proposals per rolling day across all sources. ### API key permissions Writes, including create, update, delete, and case edits, require an API key with `EVALS:WRITE`. `getCoverage` and `decomposeCriteria` also require `EVALS:WRITE` because they can call a model. Other reads accept `EVALS:READ`, `EVALS:WRITE`, `FLOWS:READ`, or `FLOWS:EXECUTE`. The run endpoint accepts `FLOWS:EXECUTE` or `EVALS:WRITE`. A change to `name`, `description`, or `graders` through `suites.update` sets the suite's `lastModifiedSource` to `sdk`, so the dashboard shows the **Managed in code** badge. The same kind of change saved in the dashboard removes the badge. Case edits don't affect it. ## Wire protocol (direct API use) If you don't use the TypeScript SDK, use two endpoints for the gate, `ensure` and `run`, plus two endpoints for reading and reviewing scores: 1. `POST /v1/eval/ensure`: Converge the suite. On the first call, send `name` and the full `definition`. `definition.name` must equal `name`. Store the `contentHash` from the response, and send it with `name` on later probes. If the hash doesn't match the saved suite, the API returns HTTP `200` with `result: "definitionRequired"`, and you retry with the full `definition`. A `contentHash` that doesn't match the `definition` you send returns `422`. A successful response includes `result`, `suiteId`, and `contentHash`. 2. `POST /v1/eval/run`: Run and score the suite. Send `suiteId` to run a saved suite, or send `definition` to run an inline virtual suite. Add `strict: true` to fail the suite on soft grader misses, and `modelOverride` to run every case against another model. Saved-suite runs save scores by default. Add `virtual: true` to skip saving. Inline definition runs are always ephemeral. 3. `GET /v1/eval/runs/{runId}/scores`: Read scores for a saved run. The response uses the same shape as the run response, plus `strict`. The API reproduces verdicts under the strict mode used to grade the run. In the SDK, call `Runtype.evals.getRunScores(runId)`. 4. `POST /v1/eval/scores/{scoreId}/review`: Record a human review of an AI-grader verdict. Send `verdict` as `agree`, `disagree`, or `null`. A `null` value clears an earlier review. A check-grader row returns HTTP `400` because its verdict is deterministic. Reviews contribute to the suite's `judgeAgreement` counts, which `GET /v1/eval/suites/{id}` returns. In the SDK, call `Runtype.evals.reviewScore(scoreId, verdict)`. The run response and the scores read include the following suite fields: | Field | Description | | ---------------------- | --------------------------------------------------------------------------- | | `runId` | The run's ID. It's `null` for a run that wasn't saved. | | `suiteId`, `name` | The suite that ran | | `targetType` | `flow` or `agent`. The run response only. | | `score` | The suite score, taken over the cases that weren't aborted by Runtype | | `passed` | Whether the suite passed. The run response only. | | `totalCases` | The number of cases in the run | | `passedCases` | The number of cases that passed | | `platformAbortedCases` | The number of cases that failed because of a Runtype model-provider timeout | | `cases` | The per-case results | Each entry in `cases` includes `name`, `passed`, `outcomes`, `outputExcerpt`, and `error`, which holds the failure text for an errored case. The run response also includes `errored`, and the scores read includes `caseId`. An errored case carries `failureKind: "platform_stall"` when it failed because of a Runtype model-provider timeout, not because of your target. Such a case runs no graders and is excluded from `score`. To find the number of cases that `score` covers, subtract `platformAbortedCases` from `totalCases`. Each outcome includes `severity` as `gate` or `soft`. An absent value means `gate`. Mark a grader soft with `severity: "soft"` in its configuration. In the scores read, each outcome also includes `scoreId` and `humanVerdict`, which can be `agree`, `disagree`, or `null`. The run response omits these fields. The `ensure` call requires `EVALS:WRITE`. The `run` call accepts `FLOWS:EXECUTE` or `EVALS:WRITE`. The scores read accepts `FLOWS:READ`, `FLOWS:EXECUTE`, `EVALS:READ`, or `EVALS:WRITE`. Reviewing a score accepts `FLOWS:EXECUTE` or `EVALS:WRITE`. Each run counts once against the daily eval limit. Reading scores and reviewing verdicts don't count. A server-side error before the API returns a result releases the run's quota slot. ## Troubleshooting The following problems stop a suite before it's graded. When `runtype eval run` hits one of them, it exits with code `2`: | Symptom | Cause | Fix | | --------------------------------------------------------------------- | ------------------------------------------------------------------ | ------------------------------------------------------------------------------------- | | `Flow named "..." was not found` or `Agent named "..." was not found` | No Flow or Agent in the account has the target name. | Create or converge the target before the gate runs, or fix the name in `target`. | | `Flow reference "..." is ambiguous` | More than one Flow, or more than one Agent, has the target name. | Rename one of them so the name matches a single target. | | `Cannot ensure a virtual eval` | `Runtype.evals.ensure` received a definition with `virtual: true`. | Remove `virtual: true` to save the suite, or don't call `ensure` for a virtual suite. | | HTTP `402` with `code: "DAILY_EVAL_LIMIT"` | The account used its daily eval runs. | Wait for the limit to reset at midnight UTC, or run fewer suites per build. | | HTTP `403` | The API key lacks `EVALS:WRITE`. | Use an API key with the `EVALS:WRITE` scope. | ## Next steps Continue with these pages: * [What are Evals?](/user-guide/evals/what-are-evals): learn the concepts behind suites, cases, and graders * [Designing evals that work](/user-guide/evals/designing-evals-that-work): choose graders and cases that catch real failures * [Interpreting eval results](/user-guide/evals/interpreting-eval-results): read a failing run and decide what to change * [Managing eval suites](/user-guide/evals/managing-eval-suites): work with saved suites on the dashboard's Evals page * [Running an eval](/user-guide/evals/running-an-eval): run and compare evals from the dashboard * [Manage Flows as code](/user-guide/flows/manage-flows-as-code): converge the Flows that your evals target * [Manage Agents as code](/user-guide/agents/manage-agents-as-code): converge the Agents that your evals target * [Run evals from the API, MCP, and CLI](/user-guide/evals/evals-via-api-and-cli): stream comparisons, submit batch evals, and pull suites without the dashboard