Run eval suite synchronously

POST

/v1/eval/run

POST

/v1/eval/run

$ curl -X POST https://api.runtype.com/v1/eval/run \
>      -H "Authorization: <apiKey>" \
>      -H "Content-Type: application/json" \
>      -d '{}'

1 {
2   "cases": [
3     {
4       "errored": false,
5       "name": "Greeting response test",
6       "outcomes": [
7         {
8           "graderIndex": 0,
9           "kind": "content_match",
10           "passed": true,
11           "evidence": [
12             {
13               "index": 0,
14               "kind": "message"
15             }
16           ],
17           "reasoning": "The response included the expected greeting phrase.",
18           "score": 1,
19           "severity": "gate",
20           "verdict": "pass"
21         }
22       ],
23       "outputExcerpt": "Hello! How can I assist you today?",
24       "passed": true
25     }
26   ],
27   "name": "Basic Greeting Suite",
28   "passed": true,
29   "passedCases": 1,
30   "runId": "a3f47b2e-9c4d-4f1a-8b2e-3d5f7a9c1b2d",
31   "score": 1,
32   "suiteId": "greeting-suite-2024",
33   "targetType": "flow",
34   "totalCases": 1
35 }

Run every case of an eval suite against its target flow/agent and grade the outputs with the suite graders, returning the suite score + per-case gate outcomes in one response. Powers the runtype eval CI gate. Saved-suite runs persist by default: the run gets a durable id (runId) and per-case scores queryable via GET /eval/runs/{runId}/scores; pass virtual: true to skip persistence. Inline definition runs are always ephemeral. Bounded — suites over the case limit must use the batch path (/eval/submit). Flow and standard-agent targets are supported (in both saved and inline/virtual modes); external and claude_managed agent targets are not.

Run every case of an eval suite against its target flow/agent and grade the outputs with the suite graders, returning the suite score + per-case gate outcomes in one response. Powers the `runtype eval` CI gate. Saved-suite runs persist by default: the run gets a durable id (`runId`) and per-case scores queryable via GET /eval/runs/{runId}/scores; pass `virtual: true` to skip persistence. Inline definition runs are always ephemeral. Bounded — suites over the case limit must use the batch path (/eval/submit). Flow and standard-agent targets are supported (in both saved and inline/virtual modes); external and claude_managed agent targets are not.

Authentication

AuthorizationBearer

API key or Clerk session token

Request

This endpoint expects an object.

definitionobjectOptional

strictbooleanOptional

Strict mode (the runtype eval --strict gate): a soft grader miss fails its case like a gate miss. Default false — soft misses are reported per-outcome but do not fail the suite.

suiteIdstringOptional>=1 character

virtualbooleanOptional

Skip persistence for a saved-suite run: no run row, no persisted scores. Saved-suite runs persist by default; inline definition runs are always ephemeral regardless of this flag.

Response

Suite score + per-case grader outcomes

caseslist of objects

namestring

passedboolean

True when every case passed every grader.

passedCasesinteger

runIdstring or null

The persisted run id — query its scores via GET /eval/runs/{runId}/scores. Null for an ephemeral (virtual/inline) run.

scoredouble

Suite score, 0..1 (passed cases / total cases).

suiteIdstring or null

targetTypeenum

totalCasesinteger

Errors

400

Bad Request Error

401

Unauthorized Error

402

Payment Required Error

403

Forbidden Error

404

Not Found Error

500

Internal Server Error