Skip to navigation
Runtype APIEvals

Run eval suite synchronously

Run every case of an eval suite against its target flow/agent and grade the outputs with the suite graders, returning the suite score + per-case gate outcomes in one response. Powers the runtype eval CI gate. Saved-suite runs persist by default: the run gets a durable id (runId) and per-case scores queryable via GET /eval/runs/{runId}/scores; pass virtual: true to skip persistence. Inline definition runs are always ephemeral. Bounded — suites over the case limit must use the batch path (/eval/submit). Flow and standard-agent targets are supported (in both saved and inline/virtual modes); external and claude_managed agent targets are not.

Authentication

AuthorizationBearer
API key or Clerk session token

Request

This endpoint expects an object.
agentobjectOptional

Version selector for an Agent target. Supply at most one of alias or versionId; the run resolves it once and every case executes that version.

definitionobjectOptional
modelOverridestringOptional>=1 character

Run every case against this model instead of the target’s configured one, without editing the suite, the flow, or the agent. Re-run the same suite once per candidate model and compare the scores to choose a model on graded evidence. Applies to flow and agent targets alike.

strictbooleanOptional

Strict mode (the runtype eval --strict gate): a soft grader miss fails its case like a gate miss. Default false — soft misses are reported per-outcome but do not fail the suite.

suiteIdstringOptional>=1 character
virtualbooleanOptional

Skip persistence for a saved-suite run: no run row, no persisted scores. Saved-suite runs persist by default; inline definition runs are always ephemeral regardless of this flag.

Response

Suite score + per-case grader outcomes

caseslist of objects
namestring
passedboolean
True when every case passed every grader.
passedCasesinteger
platformAbortedCasesinteger
Cases the platform aborted before the target could answer. They run no graders and are counted in neither half of score, so totalCases minus this is the denominator score was taken over.
runIdstring or null

The persisted run id — query its scores via GET /eval/runs/{runId}/scores. Null for an ephemeral (virtual/inline) run.

scoredouble

Suite score, 0..1 (passed cases / graded cases). Platform-aborted cases are excluded from both halves.

suiteIdstring or null
targetTypeenum
Allowed values:
totalCasesinteger
agentTargetAliasstring or nullOptional
The release alias the run resolved through, when it resolved through one.
agentTargetResolutionenumOptional
How the run picked its version. Null on a run that predates release aliases.
Allowed values:
agentVersionIdstring or nullOptional
The exact Agent version every case of this run executed. Null for a legacy run.
caseManifestHashstring or nullOptional

SHA-256 over the ordered, normalized case set this run executed.

evaluatorFingerprintstring or nullOptional

SHA-256 over the evaluator configuration this run scored with.

Errors

400
Bad Request Error
401
Unauthorized Error
402
Payment Required Error
403
Forbidden Error
404
Not Found Error
500
Internal Server Error
503
Service Unavailable Error