Run eval suite

POST

/v1/eval/suites/:id/run

POST

/v1/eval/suites/:id/run

$ curl -X POST https://api.runtype.com/v1/eval/suites/id/run \
>      -H "Authorization: <apiKey>" \
>      -H "Content-Type: application/json" \
>      -d '{}'

1 {
2   "cases": [
3     {
4       "errored": true,
5       "name": "string",
6       "outcomes": [
7         {
8           "graderIndex": 1,
9           "kind": "string",
10           "passed": true,
11           "evidence": [
12             {
13               "index": 1,
14               "kind": "message"
15             }
16           ],
17           "reasoning": "string",
18           "score": 1.1,
19           "severity": "gate",
20           "verdict": "pass"
21         }
22       ],
23       "outputExcerpt": "string",
24       "passed": true,
25       "error": "string"
26     }
27   ],
28   "name": "string",
29   "passed": true,
30   "passedCases": 1,
31   "runId": "string",
32   "score": 1.1,
33   "suiteId": "string",
34   "targetType": "flow",
35   "totalCases": 1,
36   "agentTargetAlias": "string",
37   "agentTargetResolution": "alias",
38   "agentVersionId": "string",
39   "caseManifestHash": "string",
40   "evaluatorFingerprint": "string"
41 }

Run every enabled case of the suite against its target and grade the outputs with the suite graders. Suites within the synchronous case limit (50) run inline and return the full scored result; larger suites are queued as a durable run (202) that is graded when it completes — poll GET /eval/runs/{runId}/scores. Both paths persist per-case scores to the same run history.

Authentication

AuthorizationBearer

API key or Clerk session token

Path parameters

idstringRequired>=1 character

Request

This endpoint expects an object.

agentobjectOptional

Version selector for an Agent-target suite. Supply at most one of alias or versionId; omit it to run the saved configuration.

modeenumOptional

Force the execution path. Default: sync for suites within the synchronous case limit (50), batch above it.

Allowed values:

modelOverridestringOptional>=1 character

Run every case against this model instead of the target’s configured one, without editing the suite or its target. Re-run the same suite once per candidate model and compare scores to pick a model on graded evidence. Applies to both flow and agent targets.

strictbooleanOptional

Strict grading: a soft grader miss fails its case like a gate miss.

Response

Suite ran synchronously — score + per-case grader outcomes

caseslist of objects

namestring

passedboolean

True when every case passed every grader.

passedCasesinteger

runIdstring or null

The persisted run id — query its scores via GET /eval/runs/{runId}/scores. Null for an ephemeral (virtual/inline) run.

scoredouble

Suite score, 0..1 (passed cases / total cases).

suiteIdstring or null

targetTypeenum

totalCasesinteger

agentTargetAliasstring or nullOptional

The release alias the run resolved through, when it resolved through one.

agentTargetResolutionenumOptional

How the run picked its version. Null on a run that predates release aliases.

agentVersionIdstring or nullOptional

The exact Agent version every case of this run executed. Null for a legacy run.

caseManifestHashstring or nullOptional

SHA-256 over the ordered, normalized case set this run executed.

evaluatorFingerprintstring or nullOptional

SHA-256 over the evaluator configuration this run scored with.

Errors

400

Bad Request Error

401

Unauthorized Error

402

Payment Required Error

403

Forbidden Error

404

Not Found Error

500

Internal Server Error