Manage evals as code

Keep your eval suites in your repository — next to the Flow or Agent they test — and run them as a CI gate so a regression stops in the pipeline instead of in production. This is the eval counterpart of managing Flows as code and managing Agents as code: the same ensure protocol to converge a suite, plus a run verb that executes and grades it.

Define the suite

defineEval is a pure, local constructor. It validates the shape and returns a canonical definition. A suite names a target (a saved Flow or Agent, by portable name), a set of cases (each an input plus what to check), and the graders that score every case.

1
import { defineEval, contains, judges } from '@runtypelabs/sdk'
2
3
export default defineEval({
4
  target: { flow: 'support-triage' },
5
  graders: [contains('ticket')],
6
  cases: [
7
    {
8
      name: 'billing routes to finance',
9
      input: { variables: { message: 'I was double charged' } },
10
      expect: [contains('finance'), judges.answersQuestion()],
11
    },
12
  ],
13
})

Save this as a *.eval.ts file anywhere in your repo (for example evals/support-triage.eval.ts) with the suite as the default export. The suite’s identity is its name plus your account scope; when you omit name it defaults to flow:<target> or agent:<target>, so one suite per target needs no explicit name. Give two suites for the same target distinct names (for example smoke and regression) to keep them separate.

Graders

Graders are checks run against each case’s final output. Deterministic checks are free and instant; AI judges call a model.

Suite-level graders run on every case; a case’s own expect graders are merged in after them. A case’s effective grader set is that combined list, and it is scored by exactly those graders, so cases in the same suite can check different things. Put the checks every case shares in the suite-level graders, and give each case the expect graders specific to its input.

1
export default defineEval({
2
  target: { agent: 'support-agent' },
3
  graders: [noError()], // runs on every case
4
  cases: [
5
    {
6
      name: 'refund request issues a refund',
7
      input: { variables: { message: 'I want a refund for order #1002' } },
8
      expect: [calledTool('issue_refund'), judges.answersQuestion()],
9
    },
10
    {
11
      name: 'greeting stays a plain reply',
12
      input: { variables: { message: 'hi there' } },
13
      expect: [usedNoTools(), contains('help')],
14
    },
15
  ],
16
})

Here the refund case is scored by noError, calledTool('issue_refund'), and answersQuestion, while the greeting case is scored by noError, usedNoTools, and contains('help'): two genuinely different grader sets in one suite.

AI judge grading

A judge grader watches the whole run, not just its final output. When the run’s execution trace was captured, the judge sees the steps that ran and every tool call (with inputs and outputs) alongside the final output, so process criteria like “confirms the order number before issuing a refund” or “never calls delete_record” are judged directly — write what success looks like and the criteria decides which evidence matters. For conversation-seeded cases (an input with messages), the judge is also shown the seeded conversation the run was replying to, so criteria can reference what the dialogue said without restating it in the expected answer. When the case also runs deterministic checks (for example contains or calledTool), the judge is handed each check’s pass/fail outcome as settled ground truth and told not to re-decide those structural facts itself — so write your judge criteria for the qualitative residue a check can’t express (tone, faithfulness, reasoning quality), not for a fact a check already covers. (When you need an exact, deterministic sequence assertion, use the trace graders instead.)

The judge returns one of three verdicts:

• Pass — the criteria is clearly met.
• Fail — the criteria is clearly not met.
• Insufficient evidence — the judge did not have enough information to decide. This renders as a warning and never fails the case (not even under --strict), so an unjudgeable case can’t silently block a pipeline.

Every verdict comes with the judge’s reasoning and, in the dashboard’s run results, evidence chips linking to the specific steps or tool calls the verdict rests on.

The platform judge is a Runtype-owned system flow: your criteria is embedded into a hand-crafted evaluation prompt and graded by a fast default model with an automatic cross-provider fallback. The default judge is also family-aware: when it comes from the same creator family as the model that produced the output under test, the platform substitutes a comparable judge from a different family to avoid self-preference bias (and keeps the default when no usable cross-family judge is available to your account). Set model on a judge grader to pin a judge model; a pinned model is always used exactly as written, regardless of family routing, and judges with a judgeFlowId are likewise unaffected.

Custom judge flows

When the platform judge prompt isn’t enough (for example, your verdict needs a retrieval step, a rubric lookup, or multi-step deliberation), set judgeFlowId to run one of your own flows as the judge:

1
judge('The answer follows the current returns policy.', { judgeFlowId: 'flow_abc123' })

The flow runs in your account through the normal flow pipeline (metered like any other run) and is held to a fixed contract:

• Inputs: it receives criteria, caseName, expected, and transcript as flow variables. These are the exact values the platform judge sees: {{transcript}} carries the indexed run trajectory and final output, and for conversation-seeded cases it opens with an unindexed “Seeded conversation” section (the dialogue the run was replying to). When the case also has check graders, {{transcript}} includes an unindexed “Deterministic check results” section (each check’s ground-truth pass/fail outcome) before the final output.
• Output: its final output must be the platform verdict JSON: {"reasoning": string, "verdict": "pass" | "fail" | "insufficient_evidence", "evidence": [{"kind": "message" | "tool_call", "index": number}]}.

The contract is validated when the grader runs. A miswired judge flow (missing, failing, or returning output that isn’t the verdict JSON) resolves to an insufficient-evidence warning that names the problem, so it can neither fail nor silently pass your cases. model is ignored when judgeFlowId is set; the flow’s own steps pin their models.

Splitting a big criterion into focused checks

A criterion that bundles several obligations (“confirms the order number before issuing a refund, and never promises a delivery date”) grades better as several focused judges: each verdict becomes a clear yes or no, and a failure tells you exactly which obligation broke. decomposeCriteria proposes that split for you:

1
const { subChecks } = await Runtype.evals.decomposeCriteria(
2
  'Confirms the order number before issuing a refund, and never promises a delivery date.'
3
)
4
const graders = subChecks.map((s) => judge(s.criteria))

The proposal is yours to review: edit the wording, drop what you don’t want, and save each accepted sub-check as its own judge row. Nothing is persisted by the call itself, and the resulting graders are ordinary AI graders (family-aware judge routing, .soft(), .atLeast() all apply as usual). A single returned sub-check means the criterion is already focused.

Trace graders

Output graders check what the run said. Trace graders check what the run did: which tools and steps ran, in what order, whether it completed, and what it cost. They are deterministic and free, scored against the run’s captured execution trace. Reach for them when the assertion is about behavior, not text (for example, “the agent called send_email exactly once and never called delete_record”).

1
import { defineEval, calledTool, notCalledTool, maxToolCalls, completed } from '@runtypelabs/sdk'
2
3
export default defineEval({
4
  target: { agent: 'support-agent' },
5
  graders: [completed()],
6
  cases: [
7
    {
8
      name: 'refund request uses the refund tool, not the delete tool',
9
      input: { messages: [{ role: 'user', content: 'I want a refund for order 123' }] },
10
      expect: [calledTool('issue_refund'), notCalledTool('delete_order'), maxToolCalls(3)],
11
    },
12
  ],
13
})

For ranStep and stepOrder, a step matches by its configured name, by its step id from the flow definition, or by its type — so ranStep('prompt') asserts that any prompt step ran, and stepOrder(['Draft reply', 'send-email']) mixes a prompt name with a context-step type. The trace lists steps in true execution order, so stepOrder is reliable for interleaved flows (for example prompt → upsert-record → prompt).

Trace graders are captured on the synchronous run path (the runtype eval run CI gate and client.evals.runSuite). The eval run records every executed step and tool call, including steps that disable streaming to end users (nothing is streamed to a client during an eval run). When a run produces no trace, trace graders fail with an explanatory reason rather than passing silently.

Severity: gate vs soft

By default every grader is a hard gate — a single miss fails its case. Mark a grader soft with .soft() and a miss becomes tracked-but-not-failing: it is still reported, but it no longer fails the case unless the run is strict (see --strict below). Use soft for signals you want to watch without blocking a merge — a latency budget, a cost ceiling, or a subjective judge — while keeping correctness checks as gates.

1
import { defineEval, contains, judge, latency } from '@runtypelabs/sdk'
2
3
export default defineEval({
4
  target: { flow: 'support-triage' },
5
  cases: [
6
    {
7
      name: 'billing routes to finance',
8
      input: { variables: { message: 'I was double charged' } },
9
      expect: [
10
        contains('finance'), // a gate — a miss fails the case
11
        latency(3000).soft(), // soft — over budget is reported, not blocking
12
        judge('The reply is empathetic.').soft(), // soft, requires a pass verdict
13
      ],
14
    },
15
  ],
16
})

The handles chain off any grader builder: .gate() (the default) and .soft() set severity. A soft miss shows up in the run output as a warning so the signal stays visible. (.atLeast(n) remains supported for backward compatibility, but the judge now returns a binary pass/fail verdict — mapped internally to 5/1 — so any cutoff from 2 to 5 simply requires a pass verdict, and .atLeast(1) always passes. New suites can omit it.)

Converge at deploy time

ensure makes the platform’s saved suite match your repo’s definition — creating, updating, or doing nothing — and never runs anything. Use it from CI/CD, the same place you deploy the rest of your application.

1
import { Runtype } from '@runtypelabs/sdk'
2
3
Runtype.configure({ apiKey: process.env.RUNTYPE_API_KEY })
4
5
const result = await Runtype.evals.ensure(suite)
6
// → { result: 'unchanged' | 'created' | 'updated', suiteId, contentHash }

ensure is hash-first: in the steady state it sends one tiny probe carrying only the name and a content hash, and writes nothing. When the content differs, it upserts the suite and replaces its cases. The target Flow or Agent is resolved by name, so the same definition converges every environment; which one you write is decided by the credentials you run with.

Colocate evals with a flow

You can attach eval suites to a flow definition with an inline evals array, so converging the flow converges its evals in the same call. An inline eval that omits target defaults to the flow it is attached to.

1
import { defineFlow, contains } from '@runtypelabs/sdk'
2
3
const supportTriage = defineFlow({
4
  name: 'support-triage',
5
  steps: [
6
    /* ... */
7
  ],
8
  evals: [
9
    {
10
      cases: [
11
        {
12
          name: 'billing routes to finance',
13
          input: { variables: { message: 'I was double charged' } },
14
          expect: [contains('finance')],
15
        },
16
      ],
17
    },
18
  ],
19
})
20
21
const result = await Runtype.flows.ensure(supportTriage)
22
// → { result, flowId, versionId, contentHash, evals: [{ result, suiteId, contentHash }] }

The flow and each inline suite converge through their own endpoints (flows.ensure never carries eval content on the flow wire), so the flow’s content hash is unchanged by its inline evals. flows.ensure returns each suite’s converge outcome under evals. To run a colocated suite, use its returned suiteId with client.evals.runSuite({ suiteId }), or review it in the dashboard — runtype eval run discovers standalone *.eval.ts files, not suites converged inline with a flow.

Run as a CI gate

The runtype eval run CLI command discovers your *.eval.ts files, converges each suite, runs every case against its target, grades the outputs, and maps the result to an exit code — so a regression fails the build.

$
runtype eval run                       # discover **/*.eval.ts, ensure, run, score
$
runtype eval run support-triage        # filter by id, directory prefix, or suite name
$
runtype eval run --strict              # soft grader misses fail the build too
$
runtype eval run --virtual             # run inline without persisting a suite
$
runtype eval run --junit .runtype/junit.xml
$
runtype eval run --url https://api.runtype-staging.com
$
runtype eval run --cwd packages/support-bot   # scope *.eval.ts discovery to a subdirectory

The exit code mirrors what CI expects:

Pass --strict to treat soft graders as gates, so a soft miss also fails the build. Without it, soft misses are printed as warnings but do not change the exit code.

A typical pipeline converges your Flows/Agents, then runs the gate against staging:

1
- run: runtype flows ensure
2
- run: runtype eval run --junit .runtype/junit.xml --url ${{ env.STAGING_API }}

A non-zero exit blocks the merge. Because run converges the suite definition first (via ensure), the suite stays visible in the dashboard — the payoff over a purely ephemeral --virtual gate, which persists nothing. Saved-suite gate runs also persist their scores by default: each run is recorded with a durable run id (printed by the CLI and returned as runId), and every case’s grader outcomes are saved, building run-by-run history you can read back with GET /v1/eval/runs/{runId}/scores. Pass virtual: true on the API request (or run the suite inline with --virtual) to keep a run ephemeral.

Run inline without persisting

Mark a suite virtual: true (or pass --virtual) to run it without converging anything to the dashboard — nothing is persisted, and ensure is skipped. This is the ephemeral mode for quick local checks.

Manage suites imperatively

ensure is declarative: your repo owns the suite and the platform converges to it. When you instead want to inspect suites, edit test cases, or manage a suite that is not in a repo, use the id-addressed CRUD surface — the client.evals.suites.* namespace in the SDK, backed by the /v1/eval/suites REST family.

1
import { Runtype, contains } from '@runtypelabs/sdk'
2
3
Runtype.configure({ apiKey: process.env.RUNTYPE_API_KEY })
4
5
// Create a suite attached to a flow, with initial cases
6
const suite = await Runtype.evals.suites.create({
7
  name: 'smoke',
8
  flowId: 'YOUR_FLOW_ID',
9
  graders: [contains('finance')],
10
  cases: [
11
    {
12
      name: 'billing routes to finance',
13
      input: { variables: { message: 'I was double charged' } },
14
    },
15
  ],
16
})
17
18
// Grow the suite over time
19
await Runtype.evals.suites.addCases(suite.id, [
20
  { name: 'refund request', input: { variables: { message: 'I want a refund' } } },
21
])
22
23
// Run it — sync within the case limit, queued above it
24
const result = await Runtype.evals.suites.run(suite.id, { strict: true })
25
if ('mode' in result) {
26
  console.log(`queued as durable run ${result.runId}`) // graded on completion
27
} else {
28
  console.log(`score ${result.score} (${result.passedCases}/${result.totalCases})`)
29
}

The full namespace maps one-to-one onto the REST endpoints:

A suite’s target is immutable — create a new suite to evaluate a different flow or agent.

Replay mode: serving toolMocks. By default a captured (saved_from_run) case grades the agent’s next action only (recordedToolMode: 'next_step', described above). Set the suite’s recordedToolMode to 'continue' (on defineEval, suites.create, or suites.update) to instead serve toolMocks by tool name and let the loop run past the first action, so graders can judge what the agent does after a tool result comes back (for example, “does it apologize after a failed refund?”). A replay never calls the real tool: it also works for a toolType: 'local' tool, which would otherwise pause the run on its first call waiting for a client to resolve it. When a replayed call has no matching mock, recordedToolUnmatchedPolicy decides the outcome: 'fail' (default, divergence is the signal for a regression case) or 'stub' (a canned “tool unavailable” success so the run keeps going).

toolMocks do not require a captured case. Write them directly on a manual case’s input, no checkpoint needed:

1
import { defineEval, contains } from '@runtypelabs/sdk'
2
3
export default defineEval({
4
  target: { agent: 'product-builder' },
5
  recordedToolMode: 'continue',
6
  cases: [
7
    {
8
      name: 'proceeds past a passing product validation',
9
      input: {
10
        variables: { productName: 'Acme Widgets' },
11
        toolMocks: [{ toolName: 'validate_product', output: { valid: true, issues: [] } }],
12
      },
13
      expect: [contains('looks good')],
14
    },
15
  ],
16
})

validate_product here never runs for real: the mock’s output is served in its place, so the case can grade what the agent does next instead of stalling on a live (or local, client-resolved) tool. Editing a mock’s output, on a captured case or a hand-authored one, turns it into a what-if scenario for free (for example, change a lookup result to not_found, or flip isError: true to see how the agent recovers). Continue mode, whether the mocks are captured or hand-authored, needs an agent-target suite.

Running by id. Suites within the synchronous case limit (50 enabled cases) run inline and return the same scored shape as POST /v1/eval/run. Larger suites return 202 with { mode: 'batch', runId, ... }: a queued durable run that is graded when it completes — poll GET /v1/eval/runs/{runId}/scores. Pass mode to force either path. Both persist per-case scores to the same run history, and each run counts once against your daily eval limit. Conversation-seeded cases (an input with messages) run on both paths: on a batch run each case’s seeded conversation rides its own record, and the judge sees it the same way it does on a synchronous run. Replay-mode cases are the exception: captured (checkpoint) cases, and cases whose toolMocks would be served under recordedToolMode: 'continue', run on the synchronous path only; a batch run containing one is rejected with REPLAY_MODE_CASES_UNSUPPORTED_ON_BATCH.

Machine-proposed cases go through a review queue. Generated cases never enter a suite directly. generateCases fans the target’s definition out over an explicit diversity matrix (happy path / edge case / ambiguous / adversarial / long input / formatting stress, crossed with user personas such as “frustrated customer” or “non-native speaker”), runs a gradeability check on every candidate (would two impartial experts agree on what passing looks like? Inconsistent cases are dropped, reported in droppedCount), and writes the survivors as proposals. Each proposal carries its provenance (source, sourceRefs with the assigned category and persona, and a one-sentence rationale) and waits in the queue until you acceptProposal (verbatim or with an edited body) or rejectProposal. Accepted proposals backlink the created case via acceptedCaseId. The coverage meter (getCoverage) reports which target tools and instruction clauses have no exercising case or grader; pass one gap back as generateCases(suiteId, { gap: { toolName } }) (or { gap: { instruction } }) to generate targeted proposals with source: 'coverage_gap'. Generation reserves one daily-eval quota slot per call, and each suite accepts at most 50 new proposals per rolling day across all sources.

Two writers, one suite. A suite’s Definition (name, description, graders) can be written by both this surface and ensure, so edits carry provenance: a Definition edit here stamps the suite’s lastModifiedSource (the dashboard shows a Managed in code badge while it is sdk or terraform), and a graders change invalidates the config-as-code content hash so the next ensure re-converges the repo definition instead of false-matching. Cases are server-authoritative data — adding or editing them changes neither provenance nor hash, so cases you author here survive routine deploys whose ensure probe returns unchanged. A repo definition change still replaces all cases at full converge.

Writes (create, update, delete, case edits) need a credential with EVALS:WRITE; reads accept any of EVALS:READ, EVALS:WRITE, FLOWS:READ, or FLOWS:EXECUTE; the run endpoint accepts FLOWS:EXECUTE or EVALS:WRITE.

Wire protocol (direct API use)

If you are not using the TypeScript SDK, the gate is two endpoints (ensure + run), plus two reads-and-reviews companions:

1. POST /v1/eval/ensure — converge the suite. Probe with { "name": "...", "contentHash": "<sha-256>" }; a miss returns a normal 200 { "result": "definitionRequired" }, so retry with the full definition. A converged response carries { "result": "...", "suiteId", "contentHash" }.
2. POST /v1/eval/run — run + score. Send { "suiteId": "..." } to run a saved suite, or { "definition": { ... } } to run an inline (virtual) suite. Add "strict": true to fail the suite on soft grader misses too. Saved-suite runs persist their scores by default; add "virtual": true to skip persistence (inline definition runs are always ephemeral). The response is { "suiteId", "name", "targetType", "runId", "score", "passed", "totalCases", "passedCases", "cases": [{ "name", "passed", "outcomes", "outputExcerpt", "errored" }] }, where runId is the persisted run’s id (null for an ephemeral run) and each entry of outcomes carries the grader’s severity ("gate" or "soft", absent ⇒ gate). A grader is marked soft on the wire with "severity": "soft" in its config.
3. GET /v1/eval/runs/{runId}/scores — read back a persisted run’s scores. Returns the suite score plus per-case grader outcomes in the same shape as the run response ({ "runId", "suiteId", "name", "strict", "score", "totalCases", "passedCases", "cases": [{ "caseId", "name", "passed", "outcomes", "outputExcerpt" }] }), with verdicts reproduced under the strict mode the run was graded with — except each persisted outcome additionally carries scoreId (the stored score row’s id, the review handle) and humanVerdict ("agree" / "disagree" / null), fields the live run response omits since an ephemeral run has no score rows to review. In the SDK this is client.evals.getRunScores(runId).
4. POST /v1/eval/scores/{scoreId}/review — record a lightweight human review of one AI-grader verdict. Body { "verdict": "agree" | "disagree" | null } (null clears an earlier review); a check-grader’s row returns 400 since its verdict is deterministic. Reviews accumulate into the suite’s judgeAgreement fraction returned by GET /v1/eval/suites/{id}. In the SDK this is client.evals.reviewScore(scoreId, verdict).

The ensure call needs a credential with EVALS:WRITE; the run call accepts either FLOWS:EXECUTE or EVALS:WRITE; the scores read accepts any of FLOWS:READ, FLOWS:EXECUTE, EVALS:READ, or EVALS:WRITE; reviewing a score accepts either FLOWS:EXECUTE or EVALS:WRITE. Each run counts once against your daily eval limit (reading scores and reviewing verdicts do not); a run that errors server-side before returning a result releases its slot.

Next steps

• What are Evals? — the concepts behind suites, cases, and graders
• Managing eval suites — work with saved suites on the dashboard’s Evals page
• Running an eval — running and comparing evals from the dashboard
• Manage Flows as code — converge the Flows your evals target
• Manage Agents as code — converge the Agents your evals target