Manage evals as code

Keep eval suites in your repository beside the Flow or Agent that they test. Run the suites as a continuous integration (CI) gate so a regression stops in the pipeline. This page covers the eval counterpart to managing Flows as code and managing Agents as code. It uses the ensure protocol and adds the run command to run and grade suites.

Before you begin

You need the following:

• The TypeScript SDK in your repository: npm install @runtypelabs/sdk.
• The runtype CLI for the CI gate: npm install -g @runtypelabs/cli.
• A Runtype API key with the EVALS:WRITE scope. The CLI reads it from the RUNTYPE_API_KEY environment variable, or you can run runtype login. For more information, see Managing API keys.
• A saved Flow or Agent to evaluate. A suite refers to its target by name.

Define the suite

Use defineEval to create a local definition. It validates the input shape and returns a canonical definition without calling the API. A suite names a target, a set of cases, and the graders that score each case. The target is a saved Flow or Agent identified by its name.

The following definition targets a Flow and combines suite-level and case-level graders:

import { defineEval, contains, judges } from '@runtypelabs/sdk'
export default defineEval({
  target: { flow: 'support-triage' },
  graders: [contains('ticket')],
  cases: [
    {
      name: 'billing routes to finance',
      input: { variables: { message: 'I was double charged' } },
      expected: { facts: ['routes the ticket to finance'] },
      expect: [contains('finance'), judges.answersQuestion()],
    },
  ],
})

Save the definition in a *.eval.ts or *.eval.mts file, for example evals/support-triage.eval.ts. Export the suite as the file’s default export. The CLI loads these files without a build step. It skips node_modules, build output folders such as dist and build, and every folder whose name starts with a dot, so don’t put suites under a folder such as .github/.

defineEval throws an error when the definition breaks one of these rules:

• The top level accepts only name, target, graders, cases, virtual, recordedToolMode, recordedToolUnmatchedPolicy, and identityDefaults. To set a suite description, use the dashboard or suites.update.
• Each case accepts only name, input, expected, and expect. A case input also accepts endUser and tenant, the identity the case runs as.
• cases can’t be empty.
• Case names must be unique within the suite, because the name identifies the case.
• Every case needs at least one grader, from the suite graders or its own expect.

A case’s expected object holds the reference answer. It accepts text, json, and facts (a list of strings). Graders such as matchesExpected() and judges.matchesExpected() compare the output with it.

The suite’s identity combines its name and your account scope. If you omit name, it defaults to flow:<target> or agent:<target>. A target can have one unnamed suite. Give suites for the same target distinct names, such as smoke and regression, to keep them separate.

Graders

Graders run against each case’s final output. Deterministic checks run without a model. AI judges call a model.

The SDK provides the following grader builders:

The judges object provides ready-made AI judges with built-in criteria:

Suite-level graders run on every case. The case’s expect graders are added after the suite-level graders, so each case runs both lists. Put shared checks in graders, and put input-specific checks in each case’s expect list.

The following definition applies a shared check and different case-specific checks:

import { calledTool, contains, defineEval, judges, noError, usedNoTools } from '@runtypelabs/sdk'
export default defineEval({
  target: { agent: 'support-agent' },
  graders: [noError()],
  cases: [
    {
      name: 'refund request issues a refund',
      input: { variables: { message: 'I want a refund for order #1002' } },
      expect: [calledTool('issue_refund'), judges.answersQuestion()],
    },
    {
      name: 'greeting stays a plain reply',
      input: { variables: { message: 'hi there' } },
      expect: [usedNoTools(), contains('help')],
    },
  ],
})

The refund case uses noError, calledTool('issue_refund'), and judges.answersQuestion(). The greeting case uses noError, usedNoTools, and contains('help').

AI judge grading

A judge grader evaluates the whole run, not only its final output. If the run has a captured execution trace, the judge uses the recorded steps and tool calls, including their inputs and outputs, with the final output. This supports process criteria such as confirming an order number before issuing a refund or avoiding a call to delete_record.

To grade against a reference answer, set useExpected: true. The judge then receives the case’s expected object with the run:

import { judge } from '@runtypelabs/sdk'
judge('The reply gives the same refund amount as the expected answer.', { useExpected: true })

For a conversation-seeded case, the judge also receives the seeded conversation that the run answers. Write criteria that refer to that conversation without restating it in the expected answer. If deterministic checks also run, the judge receives their pass or fail outcomes as ground truth. Write judge criteria for qualitative aspects that checks can’t express, such as tone, faithfulness, and reasoning quality. Use trace graders for exact deterministic sequence assertions.

The judge returns one of three verdicts:

• Pass: The criterion is clearly met.
• Fail: The criterion is clearly not met.
• Insufficient evidence: The run doesn’t provide enough information to decide. This verdict renders as a warning and doesn’t fail the case, including under --strict.

Each verdict includes the judge’s reasoning. Run results in the dashboard include evidence chips that link to the steps or tool calls that support the verdict.

Runtype grades the run against your criteria with a default judge model. If that model’s provider is unavailable, Runtype falls back to a model from another provider.

The default judge also uses family-aware routing. If it shares a creator family with the model that produced the output, Runtype selects a comparable judge from a different family. If no usable cross-family judge is available to your account, Runtype keeps the default judge. Set model on a judge grader to pin a judge model. A pinned model is used exactly as written, regardless of family routing. Judges with judgeFlowId also bypass family routing.

Custom judge flows

When the platform judge prompt doesn’t cover your criteria, set judgeFlowId to run one of your Flows as the judge. For example, use a retrieval step, rubric lookup, or multi-step deliberation in the judge Flow.

The following grader uses a Flow ID placeholder:

import { judge } from '@runtypelabs/sdk'
judge('The answer follows the current returns policy.', {
  judgeFlowId: 'YOUR_FLOW_ID',
})

Replace YOUR_FLOW_ID with the ID of the Flow that grades the case.

Your Flow runs through the normal Flow pipeline and counts like any other run. It follows this contract:

• Inputs: The Flow receives criteria, caseName, expected, and transcript as Flow variables. The {{transcript}} value contains the indexed run trajectory and final output. For conversation-seeded cases, it starts with an unindexed Seeded conversation section. If the case has check graders, it also contains an unindexed Deterministic check results section with each check’s ground-truth outcome.
• Output: The final output must be the platform verdict JSON: {"reasoning": string, "verdict": "pass" | "fail" | "insufficient_evidence", "evidence": [{"kind": "message" | "tool_call", "index": number}]}.

The grader validates this contract when it runs. If the judge Flow is missing, fails, or returns invalid verdict JSON, the grader returns an insufficient-evidence warning that names the problem. The warning can’t fail or silently pass the case. When judgeFlowId is set, model is ignored. Each Flow step selects its own model.

Split a large criterion into focused checks

Split criteria that combine several obligations into focused judges. For example, separate confirming an order number from avoiding delivery-date promises. Each judge then returns a clear verdict for one obligation.

Use decomposeCriteria to propose the split:

import { judge, Runtype } from '@runtypelabs/sdk'
const { subChecks } = await Runtype.evals.decomposeCriteria(
  'Confirms the order number before issuing a refund, and never promises a delivery date.'
)
const graders = subChecks.map((subCheck) => judge(subCheck.criteria))

Review the proposal before you save it. Edit the wording, remove unwanted sub-checks, and save each accepted sub-check as its own judge row. The call doesn’t save anything. The resulting graders are ordinary AI graders, so family-aware routing and .soft() apply. A single returned sub-check means that the criterion is already focused.

Trace graders

Output graders evaluate the final output. Trace graders evaluate recorded behavior, including which tools and steps run, their order, completion status, and cost. Trace graders are deterministic and free. They use the captured execution trace. Use them when the assertion concerns behavior instead of text.

The SDK provides the following trace grader builders:

The following definition checks tool usage and run completion:

import { calledTool, completed, defineEval, maxToolCalls, notCalledTool } from '@runtypelabs/sdk'
export default defineEval({
  target: { agent: 'support-agent' },
  graders: [completed()],
  cases: [
    {
      name: 'refund request uses the refund tool, not the delete tool',
      input: {
        messages: [{ role: 'user', content: 'I want a refund for order 123' }],
      },
      expect: [calledTool('issue_refund'), notCalledTool('delete_order'), maxToolCalls(3)],
    },
  ],
})

For ranStep and stepOrder, a step matches by configured name, Flow step ID, or step type. For example, ranStep('prompt') checks that any prompt step ran. stepOrder(['Draft reply', 'send-email']) combines a prompt name with a context-step type. The trace records actual execution order, so stepOrder also works for interleaved Flows such as prompt, then upsert-record, then prompt.

An eval run records every step and tool call, including steps that don’t stream to end users. If a run produces no trace, trace graders fail with an explanatory reason instead of passing silently.

Severity: gate vs soft

Each grader is a gate by default. A failed gate fails its case. Mark a grader soft with .soft() to report a miss without failing the case. A soft miss fails only when the run uses --strict.

Mark latency, cost, or subjective judge checks soft when you want to track them without blocking a merge. Keep correctness checks as gates.

The following definition uses gate and soft graders:

import { contains, defineEval, judge, latency } from '@runtypelabs/sdk'
export default defineEval({
  target: { flow: 'support-triage' },
  cases: [
    {
      name: 'billing routes to finance',
      input: { variables: { message: 'I was double charged' } },
      expect: [
        contains('finance'), // gate; a miss fails the case
        latency(3000).soft(), // soft; a miss is reported
        judge('The reply is empathetic.').soft(), // soft; requires a pass verdict
      ],
    },
  ],
})

Chain .gate() or .soft() from any grader builder. .gate() is the default. A soft miss appears as a warning in the run output.

The judge returns pass or fail, so don’t use .atLeast() in new suites. Any value from 2 to 5 requires a pass verdict, and .atLeast(1) makes the grader always pass.

Converge at deploy time

Use ensure to make the saved suite match your repository definition. It creates, updates, or leaves the suite unchanged. It doesn’t run the suite. Call it from continuous integration and continuous delivery (CI/CD), alongside the rest of your deployment.

Pass your eval definition as suite to the ensure method:

import suite from './evals/support-triage.eval'
import { Runtype } from '@runtypelabs/sdk'
Runtype.configure({ apiKey: process.env.RUNTYPE_API_KEY })
const result = await Runtype.evals.ensure(suite)
// { result: 'unchanged' | 'created' | 'updated', suiteId, contentHash }

ensure first sends a probe with the suite name and content hash. If the hash matches, it writes nothing. If the content differs, it upserts the suite and replaces the cases that ensure owns. Cases that you add in the dashboard, save from a run, generate, or import stay in the suite. The target Flow or Agent resolves by name, so the same definition converges in every environment. Your API key selects the account that you update.

If a suite with the same name was last changed in the dashboard or through the API, ensure returns a 409 with the code external_modification and changes nothing. Pass onConflict: 'overwrite' to take the suite over, or run runtype eval run --overwrite-managed from the CLI. Eval suites keep no version history, so an overwrite permanently replaces edited cases that ensure owns. After a takeover, ensure marks the suite Managed in code. If your definition names a different target under the same suite name, ensure points the existing suite at the new target.

Pass dryRun: true to see the planned change without writing it. The plan returns the live remoteHash; pass it back as expectedRemoteHash to apply only if nobody changed the suite in between.

Colocate evals with a Flow

Attach eval suites to a Flow definition with an inline evals array. Converging the Flow also converges its inline evals in the same call. An inline eval without target uses the Flow that contains it.

The following Flow definition includes an inline eval suite:

import { Runtype, contains, defineFlow } from '@runtypelabs/sdk'
const supportTriage = defineFlow({
  name: 'support-triage',
  steps: [
    /* Add the Flow steps here. */
  ],
  evals: [
    {
      cases: [
        {
          name: 'billing routes to finance',
          input: { variables: { message: 'I was double charged' } },
          expect: [contains('finance')],
        },
      ],
    },
  ],
})
const result = await Runtype.flows.ensure(supportTriage)
// { result, flowId, versionId, contentHash, evals: [{ result, suiteId, contentHash }] }

Inline evals don’t change the Flow’s content hash. The response returns each suite’s converge outcome under evals. To run a colocated suite, pass its returned suiteId to Runtype.evals.runSuite({ suiteId }), or review the suite in the dashboard. The runtype eval run command discovers standalone *.eval.ts files, not suites converged inline with a Flow.

Handle dashboard edits

A suite can change outside your repository, for example when someone edits its graders in the dashboard. Decide which version wins:

• Code wins: Run ensure with onConflict: 'overwrite'. An edit made outside your repository makes a plain ensure return 409, so the overwrite writes your repository definition back, including the cases that ensure owns.
• Dashboard wins: Run pull, copy the returned definition into your *.eval.ts file, review the git diff, then commit it or discard it.

Use this call to pull the saved definition and its provenance:

const { definition, contentHash, lastModifiedSource, updatedAt } =
  await Runtype.evals.pull('smoke')

The pulled definition lists each case’s full grader set under expect. It contains only the cases that ensure owns, so cases saved from runs stay out of your repository. A suite that ensure never wrote returns its dashboard cases, and the next ensure adopts identical cases instead of duplicating them. A name that matches no suite returns 404.

Adding a case doesn’t change the content hash, and ensure never deletes a case that it doesn’t own. Editing or deleting a case that ensure owns counts as an edit outside your repository. When your repository definition changes, ensure replaces only its own cases with the cases in your definition.

Run as a CI gate

Use the runtype eval run CLI command to discover your *.eval.ts files, converge each suite, run every case against its target, grade the outputs, and return an exit code. A regression produces a non-zero exit code.

The following commands run suites, select strict grading, or scope discovery:

runtype eval run                       # discover **/*.eval.ts, ensure, run, score
runtype eval run support-triage        # filter by path prefix, filename, or suite name
runtype eval run --strict              # soft grader misses fail the build too
runtype eval run --virtual             # run inline without persisting a suite
runtype eval run --junit .runtype/junit.xml
runtype eval run --url <api-url>       # send requests to a different API base URL
runtype eval run --cwd packages/support-bot   # scope *.eval.ts discovery to a subdirectory
runtype eval run --alias <alias>       # evaluate the version a release alias points at
runtype eval run --version-id <agent-version-id>  # evaluate one exact version snapshot

Replace the following:

• <api-url>: the base URL of the Runtype API to run against, such as a self-hosted deployment
• <alias>: a release alias on the target Agent, such as pr-482
• <agent-version-id>: an Agent version ID from GET /v1/agent-versions/{agentId}

The command returns the following exit codes:

To make soft graders fail the build, pass --strict. Without it, soft misses appear as warnings and don’t change the exit code.

The following GitHub Actions steps install your dependencies and the CLI, then run the gate:

env:
  RUNTYPE_API_KEY: ${{ secrets.RUNTYPE_API_KEY }}
steps:
  - uses: actions/checkout@v4
  - run: npm ci
  - run: npm install -g @runtypelabs/cli
  - run: runtype eval run --junit .runtype/junit.xml

Your eval files import @runtypelabs/sdk, so install your repository’s dependencies before the gate runs. The API key selects the account that the gate runs against. To gate a separate environment, store that environment’s API key as its own secret. The target Flows and Agents must exist in that account before the gate runs. Converge them in an earlier step with Runtype.flows.ensure in a deploy script, or with runtype agents ensure.

A non-zero exit blocks the merge. The run command first converges the suite definition with ensure, so the saved suite remains visible in the dashboard. Saved-suite gate runs also save scores by default. Each run receives a runId, which the CLI prints and the API returns. The saved outcomes build a run-by-run history that you can read with GET /v1/eval/runs/{runId}/scores. Pass virtual: true in the API request, or pass --virtual to the CLI, to keep the run ephemeral.

Limits and supported targets

The gate runs each suite synchronously. It has these limits:

• A suite runs at most 50 cases per run.
• Case execution has a 4-minute total budget. Cases that don’t run within the budget are reported as failed with an explanatory message, and the rest of the response is still scored.
• Keep slow, multi-step suites well under 50 cases. For larger or slower suites, use a batch eval.

The gate supports Flow, standard Agent, and external Agent targets, with these differences for external Agents:

• Delegate mode: A multi-turn case is sent as one message, and tool traces come only from what the external Agent reports.
• Managed mode: Case messages stay a multi-turn conversation, and Runtype records the tool calls it makes.
• Replay cases: Cases that replay recorded tool activity don’t run against an external Agent in either mode. Each one is reported as a failed case with a message that explains why. The other cases still run.

A Claude Managed Agent isn’t supported by the gate. Use a batch eval for that target.

Batch-evaluate a delegated external Agent

Use POST /v1/eval/submit to run a batch eval against a Delegate-mode external Agent. Supply one baseline eval config and put each record’s conversation in metadata.messages:

curl https://api.runtype.com/v1/eval/submit \
  -H "Authorization: Bearer $RUNTYPE_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "agentId": "YOUR_AGENT_ID",
    "records": [
      {
        "name": "refund request",
        "metadata": {
          "messages": [
            { "role": "user", "content": "I need a refund for order 123" }
          ]
        }
      }
    ],
    "evalConfigs": [{ "evalConfigId": "baseline", "evalName": "External baseline" }]
  }'

Replace YOUR_AGENT_ID with the ID of the external Agent.

Runtype can’t see inside the remote endpoint, so this batch evaluates only the Agent’s saved remote configuration. Omit overrides (an empty object is also accepted) and options.modelOverride. Model, step, sub-Agent, Claude Managed, and advisor overrides return EXTERNAL_AGENT_OVERRIDE_NOT_APPLICABLE instead of running an unchanged baseline. The batch has no hosted-model snapshot because Runtype doesn’t control the remote model. Supplying more than one eval config returns EXTERNAL_AGENT_COMPARISON_NOT_SUPPORTED.

Run inline without persisting

Set virtual: true on a suite, or pass --virtual, to run it without converging anything to the dashboard. This mode doesn’t save the suite or its results, and it skips ensure. Use it for local checks.

Manage suites imperatively

Use ensure when your repository owns the suite definition. When you want to inspect suites, edit cases, or manage a suite outside a repository, use the SDK’s ID-addressed create, read, update, and delete (CRUD) methods in the Runtype.evals.suites namespace. These methods use the /v1/eval/suites REST API.

The following code creates a suite, adds a case, and runs the suite:

import { Runtype, contains } from '@runtypelabs/sdk'
Runtype.configure({ apiKey: process.env.RUNTYPE_API_KEY })
const suite = await Runtype.evals.suites.create({
  name: 'smoke',
  flowId: 'YOUR_FLOW_ID',
  graders: [contains('finance')],
  cases: [
    {
      name: 'billing routes to finance',
      input: { variables: { message: 'I was double charged' } },
    },
  ],
})
await Runtype.evals.suites.addCases(suite.id, [
  { name: 'refund request', input: { variables: { message: 'I want a refund' } } },
])
const result = await Runtype.evals.suites.run(suite.id, { strict: true })
if ('mode' in result) {
  console.log(`queued as batch run ${result.runId}`)
} else {
  console.log(`score ${result.score} (${result.passedCases}/${result.totalCases})`)
}

Replace YOUR_FLOW_ID with the ID of the Flow to evaluate.

In these methods, a case’s graders replace the suite graders for that case. This differs from defineEval, where a case’s expect list adds to the suite graders. To combine them, list both. Set graders: null in updateCase to go back to the suite graders.

The SDK maps the following methods to REST endpoints:

The acceptProposal method saves a proposal as a case with origin: 'generated'. A trace-sourced proposal uses origin: 'saved_from_run'. The accepted case keeps a link back to its proposal. If you pass an edited body in options.case, the method records the acceptance as edited_accepted.

The addCaseFromExecution method freezes an Agent conversation at a fork point, attaches recorded tool results as editable mocks, and saves the case with origin: 'saved_from_run'. It returns case and replayable fields. A value of false for replayable means that a tool output was truncated during capture. The capture doesn’t run the Agent again.

These methods can’t change a suite’s target. To evaluate a different Flow or Agent, create another suite, or change the target in your defineEval definition and run ensure.

Replay mode

By default, a captured saved_from_run case grades only the Agent’s next action. The default is recordedToolMode: 'next_step'.

During replay, the first tool call is recorded as intent in the trace with its name and arguments, but it isn’t run. A text reply is graded as the output. Use a captured case with an Agent-target suite. A Flow-target suite reports an actionable failure instead of running the case.

Set recordedToolMode to 'continue' in defineEval to serve toolMocks by tool name and continue after the first action. Graders can then evaluate what the Agent does after a tool result returns. For example, they can evaluate whether the Agent apologizes after a failed refund. For a suite that you manage outside a repository, send recordedToolMode in the body of POST /v1/eval/suites or PATCH /v1/eval/suites/{id}. The TypeScript types for suites.create and suites.update don’t include this field.

A replay doesn’t call the real tool. It also works with a toolType: 'local' tool, which otherwise pauses on its first call while a client resolves it. When a replayed call has no matching mock, recordedToolUnmatchedPolicy selects the outcome. Set it to 'fail' for the default failure, or to 'stub' for a canned tool unavailable success that lets the run continue.

You don’t need a captured case to use toolMocks. Add them to a manual case’s input without a checkpoint:

import { contains, defineEval } from '@runtypelabs/sdk'
export default defineEval({
  target: { agent: 'product-builder' },
  recordedToolMode: 'continue',
  cases: [
    {
      name: 'proceeds past a passing product validation',
      input: {
        variables: { productName: 'Example Organization Widgets' },
        toolMocks: [{ toolName: 'validate_product', output: { valid: true, issues: [] } }],
      },
      expect: [contains('looks good')],
    },
  ],
})

In this example, validate_product doesn’t run. The mock’s output is served instead. Edit the output on a captured or hand-authored case to test an alternative result. For example, set a lookup result to not_found or set isError to true to test recovery. Continue mode requires an Agent target.

Grade an agent that runs as an end user

An Agent with a tenancy strategy needs an end user or a tenant on every run. Give each case the identity to run as. A case’s input accepts the same endUser and tenant objects that the body of POST /v1/dispatch accepts:

• endUser: { id, email?, ... }. Other keys are passed through.
• tenant: { id, ... }. Other keys are passed through.

The suite runs as your organization, with your API key or dashboard session. Runtype trusts that caller, so it accepts the identity as given and projects it as it projects a dispatch. The case uses the same end-user record and record scope. Memory-enabled agents remain unsupported in evals. Tools receive the identity as _endUser.* and _tenant.*. An external tool that references {{_identity.token}} receives a real signed assertion for that end user. To keep the identity inside Runtype, mock the tool by name with toolMocks and set recordedToolMode: 'continue'.

To give every case the same identity, set identityDefaults on the suite. A case that sets its own endUser or tenant uses its value for that scope and the suite default for the other scope.

The following POST /v1/eval/suites body sets a default end user and overrides it in one case:

{
  "name": "orders-agent",
  "agentId": "YOUR_AGENT_ID",
  "identityDefaults": {
    "endUser": { "id": "eval-user-1", "email": "eval-user-1@example.com" },
    "tenant": { "id": "acme" }
  },
  "graders": [{ "kind": "called_tool", "name": "list_orders" }],
  "cases": [
    {
      "name": "lists open orders",
      "input": { "messages": [{ "role": "user", "content": "Show my open orders" }] }
    },
    {
      "name": "user with no orders",
      "input": {
        "messages": [{ "role": "user", "content": "Show my open orders" }],
        "endUser": { "id": "eval-user-empty" }
      }
    }
  ]
}

Replace YOUR_AGENT_ID with the ID of the Agent to evaluate. The second case runs as eval-user-empty within the acme tenant.

To change the defaults later, send identityDefaults in PATCH /v1/eval/suites/{id}. GET /v1/eval/suites/{id} returns it. To set a case’s identity, send endUser or tenant in the case input of POST /v1/eval/suites/{id}/cases or PATCH /v1/eval/suites/{id}/cases/{caseId}.

In the Runtype MCP server, pass identity_defaults to create_eval_suite or update_eval_suite. Case inputs use the same endUser and tenant keys in add_eval_cases and update_eval_case. The following add_eval_cases arguments add a case that runs as one end user:

{
  "suite_id": "YOUR_SUITE_ID",
  "cases": [
    {
      "name": "lists open orders for a second user",
      "input": {
        "messages": [{ "role": "user", "content": "Show my open orders" }],
        "endUser": { "id": "eval-user-2" }
      }
    }
  ]
}

Replace YOUR_SUITE_ID with the suite ID. In the dashboard, set the case identity under Run as end user in the case dialog, and the suite default under Identity defaults in the suite settings.

In a suite managed as code, set identityDefaults in defineEval, and set endUser or tenant in a case input. ensure updates the suite when the value changes, because identityDefaults is part of the content hash:

import { defineEval } from '@runtypelabs/sdk'
export default defineEval({
  name: 'orders-agent',
  target: { agentId: 'YOUR_AGENT_ID' },
  identityDefaults: {
    endUser: { id: 'eval-user-1' },
    tenant: { id: 'acme' },
  },
  graders: [{ kind: 'called_tool', name: 'list_orders' }],
  cases: [
    {
      name: 'user with no orders',
      input: {
        messages: [{ role: 'user', content: 'Show my open orders' }],
        endUser: { id: 'eval-user-empty' },
      },
    },
  ],
})

A case for an Agent with a tenancy strategy fails when neither the case nor the suite supplies the identity that the strategy requires. Its error names the missing identity and ends with “Add input.endUser (or input.tenant) to the case, or identityDefaults on the suite.”

You can’t pass an identity proof (identityProof) to a suite run. A proof is a short-lived credential for an untrusted caller, such as a browser. A suite runs as your organization, so it doesn’t need one, and a proof saved on a case would expire. run_eval_suite and POST /v1/eval/suites/{id}/run don’t accept an identity override. Set the identity on the case or the suite.

Run a suite by ID

Suites with up to 50 enabled cases run synchronously and return the same scored shape as POST /v1/eval/run. Larger suites return HTTP 202 with mode: 'batch' and a runId. The API queues the run and grades it when it completes. Poll GET /v1/eval/runs/{runId}/scores for the results. Pass mode to force either path.

Conversation-seeded cases, which include messages in input, run on both paths. In a batch run, each case keeps its seeded conversation, and the judge receives it as it does on a synchronous run.

Replay-mode cases are the exception. Captured cases and cases whose toolMocks are served with recordedToolMode: 'continue' run only on the synchronous path. A batch run containing one of these cases is rejected with REPLAY_MODE_CASES_UNSUPPORTED_ON_BATCH.

Both synchronous and batch runs save per-case scores in the same run history. Each run counts once against the daily eval limit.

Compare models on the same suite

To choose a model based on graded results, run the same suite once for each candidate model with modelOverride. Every case runs against that model, and the suite and its target stay unchanged:

const haiku = await Runtype.evals.suites.run('YOUR_SUITE_ID', { modelOverride: 'claude-haiku-4-5' })
const sonnet = await Runtype.evals.suites.run('YOUR_SUITE_ID', { modelOverride: 'claude-sonnet-5' })

Replace YOUR_SUITE_ID with the ID of the suite to run. Compare the scores of the two runs. POST /v1/eval/run also accepts modelOverride in its request body.

Evaluate a specific Agent version

An Agent-target run can name the version it evaluates instead of taking the saved configuration. POST /v1/eval/suites/{suiteId}/run, POST /v1/eval/run, and POST /v1/eval/submit all accept an agent selector carrying exactly one of alias or versionId:

{ "agent": { "alias": "pr-482" } }

The MCP run_eval_suite and submit_eval tools expose the same choice as agent_alias and agent_version_id. The CLI’s runtype eval run takes --alias or --version-id, and the SDK’s Runtype.evals.suites.run takes an agent selector.

The selector belongs to the run, not to the suite. A suite is a fixed case set, and you choose which version to grade it against each time you run it. The run resolves the selector once, before its first case, so moving the alias during the run doesn’t split the run across two versions.

Selecting an Agent version pins its configuration, not its Skill bindings. Agent-target suite runs resolve the Agent’s current enabled bindings once for the run. Each binding uses its pinned Skill version, or the Skill’s current published version when none is pinned. Saved suite runs record the resolved slug, skillId, and skillVersionId values in evalConfig.skillBindings. To keep Skill content constant across runs, pin the bindings to Skill versions and keep the bindings unchanged.

Every run records what it evaluated, and the run reads return it with the scores:

• agentVersionId: the version that ran.
• agentTargetResolution: alias, version, or legacy-live-row for a run that named no selector while the Agent still ran its saved configuration.
• agentTargetAlias: the alias that the run named.
• caseManifestHash: a SHA-256 hash of the case set that the run used.
• evaluatorFingerprint: a SHA-256 hash of the judges and grading configuration.

Runtype records this evidence but doesn’t enforce it. Runtype never requires a passing run before a version is deployed, and never blocks or reverses a deployment because of a failing run. A run that resolved a single target gives you the exact version ID to activate.

A run that names no selector resolves the same way as any other request to that Agent that names no version. For details, see What runs when you do not name a version.

A run fails with one of the following errors when Runtype can’t resolve the version:

When a suite targets an Agent whose live alias is archived, Runtype warns you when you save the suite, before the first run. The create, read, and update responses for that suite carry a warnings entry with code: "AGENT_NOT_DEPLOYED". Deploy a version to that Agent’s live alias to clear it.

Review generated case proposals

Generated cases don’t enter a suite directly. generateCases creates candidates from the target definition across a category and persona matrix. Categories include happy path, edge case, ambiguous, adversarial, long input, and formatting stress. Personas include a frustrated customer and a non-native speaker.

The API filters candidates for gradeability. It reports dropped candidates in droppedCount and stores the remaining candidates as proposals. Each proposal stores its source, sourceRefs with the assigned category and persona, and a one-sentence rationale. Review each proposal with acceptProposal or rejectProposal. Accept a proposal as written, or pass options.case to save an edited body. Accepted proposals link to the created case through acceptedCaseId.

The getCoverage method reports target tools and instruction clauses without an exercising case or grader. Pass one gap to generateCases, such as { gap: { toolName } } or { gap: { instruction } }, to create targeted proposals with source: 'coverage_gap'. Each generation call reserves one daily-eval quota slot. A suite accepts at most 50 new proposals per rolling day across all sources.

API key permissions

Writes, including create, update, delete, and case edits, require an API key with EVALS:WRITE. getCoverage and decomposeCriteria also require EVALS:WRITE because they can call a model. Other reads accept EVALS:READ, EVALS:WRITE, FLOWS:READ, or FLOWS:EXECUTE. The run endpoint accepts FLOWS:EXECUTE or EVALS:WRITE.

A change to name, description, or graders through suites.update sets the suite’s lastModifiedSource to sdk, so the dashboard shows the Managed in code badge. The same kind of change saved in the dashboard removes the badge. Case edits don’t affect it.

Wire protocol (direct API use)

If you don’t use the TypeScript SDK, use two endpoints for the gate, ensure and run, plus two endpoints for reading and reviewing scores:

1. POST /v1/eval/ensure: Converge the suite. On the first call, send name and the full definition. definition.name must equal name. Store the contentHash from the response, and send it with name on later probes. If the hash doesn’t match the saved suite, the API returns HTTP 200 with result: "definitionRequired", and you retry with the full definition. A contentHash that doesn’t match the definition you send returns 422. A successful response includes result, suiteId, and contentHash.
2. POST /v1/eval/run: Run and score the suite. Send suiteId to run a saved suite, or send definition to run an inline virtual suite. Add strict: true to fail the suite on soft grader misses, and modelOverride to run every case against another model. Saved-suite runs save scores by default. Add virtual: true to skip saving. Inline definition runs are always ephemeral.
3. GET /v1/eval/runs/{runId}/scores: Read scores for a saved run. The response uses the same shape as the run response, plus strict. The API reproduces verdicts under the strict mode used to grade the run. In the SDK, call Runtype.evals.getRunScores(runId).
4. POST /v1/eval/scores/{scoreId}/review: Record a human review of an AI-grader verdict. Send verdict as agree, disagree, or null. A null value clears an earlier review. A check-grader row returns HTTP 400 because its verdict is deterministic. Reviews contribute to the suite’s judgeAgreement counts, which GET /v1/eval/suites/{id} returns. In the SDK, call Runtype.evals.reviewScore(scoreId, verdict).

The run response and the scores read include the following suite fields:

Each entry in cases includes name, passed, outcomes, outputExcerpt, and error, which holds the failure text for an errored case. The run response also includes errored, and the scores read includes caseId.

An errored case carries failureKind: "platform_stall" when it failed because of a Runtype model-provider timeout, not because of your target. Such a case runs no graders and is excluded from score. To find the number of cases that score covers, subtract platformAbortedCases from totalCases.

Each outcome includes severity as gate or soft. An absent value means gate. Mark a grader soft with severity: "soft" in its configuration. In the scores read, each outcome also includes scoreId and humanVerdict, which can be agree, disagree, or null. The run response omits these fields.

The ensure call requires EVALS:WRITE. The run call accepts FLOWS:EXECUTE or EVALS:WRITE. The scores read accepts FLOWS:READ, FLOWS:EXECUTE, EVALS:READ, or EVALS:WRITE. Reviewing a score accepts FLOWS:EXECUTE or EVALS:WRITE. Each run counts once against the daily eval limit. Reading scores and reviewing verdicts don’t count. A server-side error before the API returns a result releases the run’s quota slot.

Troubleshooting

The following problems stop a suite before it’s graded. When runtype eval run hits one of them, it exits with code 2:

Next steps

Continue with these pages:

• What are Evals?: learn the concepts behind suites, cases, and graders
• Designing evals that work: choose graders and cases that catch real failures
• Interpreting eval results: read a failing run and decide what to change
• Managing eval suites: work with saved suites on the dashboard’s Evals page
• Running an eval: run and compare evals from the dashboard
• Manage Flows as code: converge the Flows that your evals target
• Manage Agents as code: converge the Agents that your evals target
• Run evals from the API, MCP, and CLI: stream comparisons, submit batch evals, and pull suites without the dashboard