Manage evals as code

Keep eval suites in your repository beside the Flow or Agent that they test. Run the suites as a continuous integration (CI) gate so a regression stops in the pipeline. This page covers the eval counterpart to managing Flows as code and managing Agents as code. It uses the ensure protocol and adds the run command to run and grade suites.

Define the suite

Use defineEval to create a pure, local definition. It validates the input shape and returns a canonical definition. A suite names a target, a set of cases, and the graders that score each case. The target is a saved Flow or Agent identified by its portable name.

The following definition targets a Flow and combines suite-level and case-level graders:

1
import { defineEval, contains, judges } from '@runtypelabs/sdk'
2
3
export default defineEval({
4
  target: { flow: 'support-triage' },
5
  graders: [contains('ticket')],
6
  cases: [
7
    {
8
      name: 'billing routes to finance',
9
      input: { variables: { message: 'I was double charged' } },
10
      expect: [contains('finance'), judges.answersQuestion()],
11
    },
12
  ],
13
})

Save the definition in a *.eval.ts file anywhere in your repository. For example, use evals/support-triage.eval.ts. Export the suite as the file’s default export.

The suite’s identity combines its name and your account scope. If you omit name, it defaults to flow:<target> or agent:<target>. A target can have one unnamed suite. Give suites for the same target distinct names, such as smoke and regression, to keep them separate.

Graders

Graders run against each case’s final output. Deterministic checks run without a model. AI judges call a model.

The SDK provides the following grader builders:

Suite-level graders run on every case. The case’s expect graders are merged after the suite-level graders. The effective grader set contains both lists, so cases in one suite can check different things. Put shared checks in graders, and put input-specific checks in each case’s expect list.

The following definition applies a shared check and different case-specific checks:

1
import { calledTool, contains, defineEval, judges, noError, usedNoTools } from '@runtypelabs/sdk'
2
3
export default defineEval({
4
  target: { agent: 'support-agent' },
5
  graders: [noError()],
6
  cases: [
7
    {
8
      name: 'refund request issues a refund',
9
      input: { variables: { message: 'I want a refund for order #1002' } },
10
      expect: [calledTool('issue_refund'), judges.answersQuestion()],
11
    },
12
    {
13
      name: 'greeting stays a plain reply',
14
      input: { variables: { message: 'hi there' } },
15
      expect: [usedNoTools(), contains('help')],
16
    },
17
  ],
18
})

The refund case uses noError, calledTool('issue_refund'), and judges.answersQuestion(). The greeting case uses noError, usedNoTools, and contains('help'). The suite therefore contains two different effective grader sets.

AI judge grading

A judge grader evaluates the whole run, not only its final output. If the run has a captured execution trace, the judge uses the recorded steps and tool calls, including their inputs and outputs, with the final output. This supports process criteria such as confirming an order number before issuing a refund or avoiding a call to delete_record.

For a conversation-seeded case, the judge also receives the seeded conversation that the run answers. Write criteria that refer to that conversation without restating it in the expected answer. If deterministic checks also run, the judge receives their pass or fail outcomes as ground truth. Write judge criteria for qualitative aspects that checks cannot express, such as tone, faithfulness, and reasoning quality. Use trace graders for exact deterministic sequence assertions.

The judge returns one of three verdicts:

• Pass: The criterion is clearly met.
• Fail: The criterion is clearly not met.
• Insufficient evidence: The run does not provide enough information to decide. This verdict renders as a warning and does not fail the case, including under --strict.

Each verdict includes the judge’s reasoning. Run results in the dashboard include evidence chips that link to the steps or tool calls that support the verdict.

The platform judge is a Runtype-owned system Flow. Runtype embeds your criteria in an evaluation prompt and grades the run with a default model. An automatic cross-provider fallback handles unavailable providers.

The default judge also uses family-aware routing. If it shares a creator family with the model that produced the output, Runtype selects a comparable judge from a different family. If no usable cross-family judge is available to your account, Runtype keeps the default judge. Set model on a judge grader to pin a judge model. A pinned model is used exactly as written, regardless of family routing. Judges with judgeFlowId also bypass family routing.

Custom judge flows

When the platform judge prompt does not cover your criteria, set judgeFlowId to run one of your Flows as the judge. For example, use a retrieval step, rubric lookup, or multi-step deliberation in the judge Flow.

The following grader uses a Flow ID placeholder:

1
import { judge } from '@runtypelabs/sdk'
2
3
judge('The answer follows the current returns policy.', {
4
  judgeFlowId: 'YOUR_FLOW_ID',
5
})

Replace YOUR_FLOW_ID with the ID of the Flow that grades the case.

Your Flow runs through the normal Flow pipeline and counts like any other run. It follows this contract:

• Inputs: The Flow receives criteria, caseName, expected, and transcript as Flow variables. The {{transcript}} value contains the indexed run trajectory and final output. For conversation-seeded cases, it starts with an unindexed Seeded conversation section. If the case has check graders, it also contains an unindexed Deterministic check results section with each check’s ground-truth outcome.
• Output: The final output must be the platform verdict JSON: {"reasoning": string, "verdict": "pass" | "fail" | "insufficient_evidence", "evidence": [{"kind": "message" | "tool_call", "index": number}]}.

The grader validates this contract when it runs. If the judge Flow is missing, fails, or returns invalid verdict JSON, the grader returns an insufficient-evidence warning that names the problem. The warning cannot fail or silently pass the case. When judgeFlowId is set, model is ignored. Each Flow step selects its own model.

Split a large criterion into focused checks

Split criteria that combine several obligations into focused judges. For example, separate confirming an order number from avoiding delivery-date promises. Each judge then returns a clear verdict for one obligation.

Use decomposeCriteria to propose the split:

1
import { judge, Runtype } from '@runtypelabs/sdk'
2
3
const { subChecks } = await Runtype.evals.decomposeCriteria(
4
  'Confirms the order number before issuing a refund, and never promises a delivery date.'
5
)
6
const graders = subChecks.map((subCheck) => judge(subCheck.criteria))

Review the proposal before you save it. Edit the wording, remove unwanted sub-checks, and save each accepted sub-check as its own judge row. The call does not persist anything. The resulting graders are ordinary AI graders, so family-aware routing, .soft(), and .atLeast() apply. A single returned sub-check means that the criterion is already focused.

Trace graders

Output graders evaluate the final output. Trace graders evaluate recorded behavior, including which tools and steps run, their order, completion status, and cost. Trace graders are deterministic and free. They use the captured execution trace. Use them when the assertion concerns behavior instead of text.

The SDK provides the following trace grader builders:

The following definition checks tool usage and run completion:

1
import { calledTool, completed, defineEval, maxToolCalls, notCalledTool } from '@runtypelabs/sdk'
2
3
export default defineEval({
4
  target: { agent: 'support-agent' },
5
  graders: [completed()],
6
  cases: [
7
    {
8
      name: 'refund request uses the refund tool, not the delete tool',
9
      input: {
10
        messages: [{ role: 'user', content: 'I want a refund for order 123' }],
11
      },
12
      expect: [calledTool('issue_refund'), notCalledTool('delete_order'), maxToolCalls(3)],
13
    },
14
  ],
15
})

For ranStep and stepOrder, a step matches by configured name, Flow step ID, or step type. For example, ranStep('prompt') checks that any prompt step ran. stepOrder(['Draft reply', 'send-email']) combines a prompt name with a context-step type. The trace records actual execution order, so stepOrder also works for interleaved Flows such as prompt → upsert-record → prompt.

Trace graders are captured on the synchronous run path, including the runtype eval run CI gate and client.evals.runSuite. The eval run records every executed step and tool call, including steps that disable streaming to end users. Nothing is streamed to a client during an eval run. If a run produces no trace, trace graders fail with an explanatory reason instead of passing silently.

Severity: gate vs soft

Each grader is a gate by default. A failed gate fails its case. Mark a grader soft with .soft() to report a miss without failing the case. A soft miss fails only when the run uses --strict.

Mark latency, cost, or subjective judge checks soft when you want to track them without blocking a merge. Keep correctness checks as gates.

The following definition uses gate and soft graders:

1
import { contains, defineEval, judge, latency } from '@runtypelabs/sdk'
2
3
export default defineEval({
4
  target: { flow: 'support-triage' },
5
  cases: [
6
    {
7
      name: 'billing routes to finance',
8
      input: { variables: { message: 'I was double charged' } },
9
      expect: [
10
        contains('finance'), // gate; a miss fails the case
11
        latency(3000).soft(), // soft; a miss is reported
12
        judge('The reply is empathetic.').soft(), // soft; requires a pass verdict
13
      ],
14
    },
15
  ],
16
})

Chain .gate() or .soft() from any grader builder. .gate() is the default. A soft miss appears as a warning in the run output. The .atLeast(n) handle remains for backward compatibility. The judge returns a binary pass or fail verdict, which maps internally to 5 or 1. Cutoffs from 2 to 5 therefore require a pass verdict. .atLeast(1) passes, and new suites can omit it.

Converge at deploy time

Use ensure to make the saved suite match your repository definition. It creates, updates, or leaves the suite unchanged. It does not run the suite. Call it from continuous integration and continuous delivery (CI/CD), alongside the rest of your deployment.

Pass your eval definition as suite to the ensure method:

1
import suite from './evals/support-triage.eval'
2
import { Runtype } from '@runtypelabs/sdk'
3
4
Runtype.configure({ apiKey: process.env.RUNTYPE_API_KEY })
5
6
const result = await Runtype.evals.ensure(suite)
7
// { result: 'unchanged' | 'created' | 'updated', suiteId, contentHash }

ensure first sends a probe with the suite name and content hash. If the hash matches, it writes nothing. If the content differs, it upserts the suite and replaces its cases. The target Flow or Agent resolves by name, so the same definition converges in every environment. Your credentials select the target that you update.

Colocate evals with a Flow

Attach eval suites to a Flow definition with an inline evals array. Converging the Flow also converges its inline evals in the same call. An inline eval without target uses the Flow that contains it.

The following Flow definition includes an inline eval suite:

1
import { Runtype, contains, defineFlow } from '@runtypelabs/sdk'
2
3
const supportTriage = defineFlow({
4
  name: 'support-triage',
5
  steps: [
6
    /* Add the Flow steps here. */
7
  ],
8
  evals: [
9
    {
10
      cases: [
11
        {
12
          name: 'billing routes to finance',
13
          input: { variables: { message: 'I was double charged' } },
14
          expect: [contains('finance')],
15
        },
16
      ],
17
    },
18
  ],
19
})
20
21
const result = await Runtype.flows.ensure(supportTriage)
22
// { result, flowId, versionId, contentHash, evals: [{ result, suiteId, contentHash }] }

The Flow and each inline suite use separate endpoints. flows.ensure does not send eval content in the Flow request, so inline evals do not change the Flow’s content hash. The response returns each suite’s converge outcome under evals. To run a colocated suite, pass its returned suiteId to client.evals.runSuite({ suiteId }), or review the suite in the dashboard. The runtype eval run command discovers standalone *.eval.ts files, not suites converged inline with a Flow.

Run as a CI gate

Use the runtype eval run CLI command to discover your *.eval.ts files, converge each suite, run every case against its target, grade the outputs, and return an exit code. A regression produces a non-zero exit code.

The following commands run suites, select strict grading, or scope discovery:

$
runtype eval run                       # discover **/*.eval.ts, ensure, run, score
$
runtype eval run support-triage        # filter by ID, directory prefix, or suite name
$
runtype eval run --strict              # soft grader misses fail the build too
$
runtype eval run --virtual             # run inline without persisting a suite
$
runtype eval run --junit .runtype/junit.xml
$
runtype eval run --url https://api.runtype-staging.com
$
runtype eval run --cwd packages/support-bot   # scope *.eval.ts discovery to a subdirectory
$
runtype eval run --alias <alias>       # evaluate the version a release alias points at
$
runtype eval run --version-id <agent-version-id>  # evaluate one exact version snapshot

Replace the following:

• <alias>: a release alias on the target Agent, such as pr-482
• <agent-version-id>: an Agent version id from GET /v1/agents/{agentId}/versions

The command returns the following exit codes:

To make soft graders fail the build, pass --strict. Without it, soft misses appear as warnings and do not change the exit code.

A typical pipeline converges your Flows and Agents, then runs the gate against staging:

1
- run: runtype flows ensure
2
- run: runtype eval run --junit .runtype/junit.xml --url ${{ env.STAGING_API }}

Set the STAGING_API environment variable to the staging API URL in your CI workflow.

A non-zero exit blocks the merge. The run command first converges the suite definition with ensure, so the saved suite remains visible in the dashboard. Saved-suite gate runs also persist scores by default. Each run receives a durable runId, which the CLI prints and the API returns. The saved outcomes build a run-by-run history that you can read with GET /v1/eval/runs/{runId}/scores. Pass virtual: true in the API request, or pass --virtual to the CLI, to keep the run ephemeral.

Batch-evaluate a delegated external Agent

Use POST /v1/eval/submit when a Delegate-mode external Agent needs the asynchronous batch lane. Supply one baseline eval config and put each record’s conversation in metadata.messages:

$
curl https://api.runtype.com/v1/eval/submit \
>
  -H "Authorization: Bearer $RUNTYPE_API_KEY" \
>
  -H "Content-Type: application/json" \
>
  -d '{
>
    "agentId": "agent_...",
>
    "records": [
>
      {
>
        "name": "refund request",
>
        "metadata": {
>
          "messages": [
>
            { "role": "user", "content": "I need a refund for order 123" }
>
          ]
>
        }
>
      }
>
    ],
>
    "evalConfigs": [{ "evalConfigId": "baseline", "evalName": "External baseline" }]
>
  }'

The remote endpoint is opaque to Runtype, so this lane evaluates only the Agent’s saved remote configuration. Omit overrides (an empty object is also accepted) and options.modelOverride. Model, step, sub-Agent, Claude Managed, and advisor overrides return EXTERNAL_AGENT_OVERRIDE_NOT_APPLICABLE instead of running an unchanged baseline. The batch has no hosted-model snapshot because Runtype does not control or observe the remote model. Supplying more than one eval config returns EXTERNAL_AGENT_COMPARISON_NOT_SUPPORTED because Runtype cannot vary the opaque endpoint between configs.

Run inline without persisting

Set virtual: true on a suite, or pass --virtual, to run it without converging anything to the dashboard. This mode does not persist the suite or its results, and it skips ensure. Use it for local checks.

Manage suites imperatively

Use ensure when your repository owns the suite definition. When you want to inspect suites, edit cases, or manage a suite outside a repository, use the SDK’s ID-addressed create, read, update, and delete (CRUD) methods in the client.evals.suites.* namespace. These methods use the /v1/eval/suites REST API.

The following code creates a suite, adds a case, and runs the suite:

1
import { Runtype, contains } from '@runtypelabs/sdk'
2
3
Runtype.configure({ apiKey: process.env.RUNTYPE_API_KEY })
4
5
const suite = await Runtype.evals.suites.create({
6
  name: 'smoke',
7
  flowId: 'YOUR_FLOW_ID',
8
  graders: [contains('finance')],
9
  cases: [
10
    {
11
      name: 'billing routes to finance',
12
      input: { variables: { message: 'I was double charged' } },
13
    },
14
  ],
15
})
16
17
await Runtype.evals.suites.addCases(suite.id, [
18
  { name: 'refund request', input: { variables: { message: 'I want a refund' } } },
19
])
20
21
const result = await Runtype.evals.suites.run(suite.id, { strict: true })
22
if ('mode' in result) {
23
  console.log(`queued as durable run ${result.runId}`)
24
} else {
25
  console.log(`score ${result.score} (${result.passedCases}/${result.totalCases})`)
26
}

Replace YOUR_FLOW_ID with the ID of the Flow to evaluate.

The SDK maps the following methods to REST endpoints:

The acceptProposal method materializes a proposal as a case with origin: 'generated'. A trace-sourced proposal uses origin: 'saved_from_run'. The accepted case keeps a provenance backlink. If you pass an edited body in options.case, the method records the acceptance as edited_accepted.

The addCaseFromExecution method freezes an Agent conversation at a fork point, attaches recorded tool results as editable mocks, and saves the case with origin: 'saved_from_run'. It returns case and replayable fields. A value of false for replayable means that a tool output was truncated during capture. The capture does not run the Agent again.

A suite’s target is immutable. Create another suite to evaluate a different Flow or Agent.

Replay mode

By default, a captured saved_from_run case grades only the Agent’s next action. The default is recordedToolMode: 'next_step'.

During replay, the first tool call is recorded as intent in the trace with its name and arguments, but it is not executed. A text reply is graded as the output. Use a captured case with an Agent-target suite. A Flow-target suite reports an actionable failure instead of running the case.

Set recordedToolMode to 'continue' on defineEval, suites.create, or suites.update to serve toolMocks by tool name and continue after the first action. Graders can then evaluate what the Agent does after a tool result returns. For example, they can evaluate whether the Agent apologizes after a failed refund.

A replay does not call the real tool. It also works with a toolType: 'local' tool, which otherwise pauses on its first call while a client resolves it. When a replayed call has no matching mock, recordedToolUnmatchedPolicy selects the outcome. Set it to 'fail' for the default failure, or to 'stub' for a canned tool unavailable success that lets the run continue.

You do not need a captured case to use toolMocks. Add them to a manual case’s input without a checkpoint:

1
import { contains, defineEval } from '@runtypelabs/sdk'
2
3
export default defineEval({
4
  target: { agent: 'product-builder' },
5
  recordedToolMode: 'continue',
6
  cases: [
7
    {
8
      name: 'proceeds past a passing product validation',
9
      input: {
10
        variables: { productName: 'Example Organization Widgets' },
11
        toolMocks: [{ toolName: 'validate_product', output: { valid: true, issues: [] } }],
12
      },
13
      expect: [contains('looks good')],
14
    },
15
  ],
16
})

In this example, validate_product does not run. The mock’s output is served instead. Edit the output on a captured or hand-authored case to test an alternative result. For example, set a lookup result to not_found or set isError to true to test recovery. Continue mode requires an Agent target.

Run a suite by ID

Suites with up to 50 enabled cases run synchronously and return the same scored shape as POST /v1/eval/run. Larger suites return HTTP 202 with mode: 'batch' and a runId. The API queues the run and grades it when it completes. Poll GET /v1/eval/runs/{runId}/scores for the results. Pass mode to force either path.

Conversation-seeded cases, which include messages in input, run on both paths. In a batch run, each case keeps its seeded conversation on its own record, and the judge receives it as it does on a synchronous run.

Replay-mode cases are the exception. Captured cases and cases whose toolMocks are served with recordedToolMode: 'continue' run only on the synchronous path. A batch run containing one of these cases is rejected with REPLAY_MODE_CASES_UNSUPPORTED_ON_BATCH.

Both synchronous and batch runs persist per-case scores in the same run history. Each run counts once against the daily eval limit.

Evaluate a specific Agent version

An Agent-target run can name the version it evaluates instead of taking the saved configuration. POST /v1/eval/suites/{suiteId}/run, POST /v1/eval/run, and POST /v1/eval/submit all accept an agent selector carrying exactly one of alias or versionId:

1
{ "agent": { "alias": "pr-482" } }

The MCP run_eval_suite and submit_eval tools expose the same choice as agent_alias and agent_version_id, and the CLI’s runtype eval run takes --alias or --version-id.

The selector belongs to the run, not to the suite: a suite is a fixed case set, and which version you grade it against is a run-time choice. The run resolves the selector once, before its first case, so moving the alias mid-run does not split the run across two definitions.

Every run records what it evaluated, and the run reads return it alongside the scores: agentVersionId, agentTargetResolution (alias, version, or legacy-live-row when no selector was supplied), agentTargetAlias, caseManifestHash (SHA-256 over the case set the run used) and evaluatorFingerprint (SHA-256 over the judges and grading configuration).

Omitting the selector keeps today’s behavior: the run executes the Agent’s saved configuration and records agentTargetResolution: "legacy-live-row". Selectors are not available for external or Claude Managed Agents, which answer 422 with code: "selector_unsupported_agent_type", and a Flow-target suite answers 400.

Review generated case proposals

Generated cases do not enter a suite directly. generateCases creates candidates from the target definition across a category and persona matrix. Categories include happy path, edge case, ambiguous, adversarial, long input, and formatting stress. Personas include a frustrated customer and a non-native speaker.

The API filters candidates for gradeability. It reports dropped candidates in droppedCount and stores the remaining candidates as proposals. Each proposal stores its source, sourceRefs with the assigned category and persona, and a one-sentence rationale. Review each proposal with acceptProposal or rejectProposal. Accept a proposal as written, or pass options.case to save an edited body. Accepted proposals link to the created case through acceptedCaseId.

The getCoverage method reports target tools and instruction clauses without an exercising case or grader. Pass one gap to generateCases, such as { gap: { toolName } } or { gap: { instruction } }, to create targeted proposals with source: 'coverage_gap'. Each generation call reserves one daily-eval quota slot. A suite accepts at most 50 new proposals per rolling day across all sources.

Manage a suite from code and the SDK

A suite definition contains name, description, and graders. Both this SDK namespace and ensure can write the definition. A definition edit in this namespace sets lastModifiedSource. The dashboard shows the Managed in code badge when that value is sdk or terraform.

Changing graders invalidates the config-as-code content hash. The next ensure call then re-converges the repository definition instead of treating it as an unchanged match. Cases remain server-authoritative data. Adding or editing a case does not change provenance or the hash, so those cases survive a deploy when the ensure probe returns unchanged. A repository definition change still replaces all cases during full convergence.

Writes, including create, update, delete, and case edits, require a credential with EVALS:WRITE. Reads accept EVALS:READ, EVALS:WRITE, FLOWS:READ, or FLOWS:EXECUTE. The run endpoint accepts FLOWS:EXECUTE or EVALS:WRITE.

Wire protocol (direct API use)

If you do not use the TypeScript SDK, use two endpoints for the gate, ensure and run, plus two endpoints for reading and reviewing scores:

1. POST /v1/eval/ensure: Converge the suite. Send a probe with the suite name and contentHash. If the hash misses, the API returns HTTP 200 with result: "definitionRequired". Retry with the full definition. A successful response includes result, suiteId, and contentHash.
2. POST /v1/eval/run: Run and score the suite. Send suiteId to run a saved suite, or send definition to run an inline virtual suite. Add strict: true to fail the suite on soft grader misses. Saved-suite runs persist scores by default. Add virtual: true to skip persistence. Inline definition runs are always ephemeral. The response includes suiteId, name, targetType, runId, score, passed, totalCases, passedCases, and cases. Each case includes name, passed, outcomes, outputExcerpt, and errored. An errored case also carries error, the provider or execution failure text, so a failing run is diagnosable from the result itself. Each outcome includes severity as gate or soft; an absent value means gate. Mark a grader soft with severity: "soft" in its configuration.
3. GET /v1/eval/runs/{runId}/scores: Read scores for a persisted run. The response includes the suite score and per-case grader outcomes. It uses the same shape as the run response and includes runId, suiteId, name, strict, score, totalCases, passedCases, and cases. Each case includes caseId, name, passed, outcomes, outputExcerpt, and error (the failure text for an errored case, otherwise null). The API reproduces verdicts under the strict mode used to grade the run. Each persisted outcome also includes scoreId and humanVerdict, which can be agree, disagree, or null. The live run response omits these fields because ephemeral runs have no score rows to review. In the SDK, call client.evals.getRunScores(runId).
4. POST /v1/eval/scores/{scoreId}/review: Record a human review of an AI-grader verdict. Send verdict as agree, disagree, or null; null clears an earlier review. A check-grader row returns HTTP 400 because its verdict is deterministic. Reviews contribute to the suite’s judgeAgreement fraction, which GET /v1/eval/suites/{id} returns. In the SDK, call client.evals.reviewScore(scoreId, verdict).

The ensure call requires EVALS:WRITE. The run call accepts FLOWS:EXECUTE or EVALS:WRITE. The scores read accepts FLOWS:READ, FLOWS:EXECUTE, EVALS:READ, or EVALS:WRITE. Reviewing a score accepts FLOWS:EXECUTE or EVALS:WRITE. Each run counts once against the daily eval limit. Reading scores and reviewing verdicts do not count. A server-side error before the API returns a result releases the run’s quota slot.

Next steps

Continue with these pages:

• What are Evals?: learn the concepts behind suites, cases, and graders
• Managing eval suites: work with saved suites on the dashboard’s Evals page
• Running an eval: run and compare evals from the dashboard
• Manage Flows as code: converge the Flows that your evals target
• Manage Agents as code: converge the Agents that your evals target