Run eval suite

POST

/v1/eval/suites/:id/run

POST

/v1/eval/suites/:id/run

$ curl -X POST https://api.runtype.com/v1/eval/suites/id/run \
>      -H "Authorization: <apiKey>" \
>      -H "Content-Type: application/json" \
>      -d '{}'

1 {
2   "cases": [
3     {
4       "errored": false,
5       "name": "Login Flow Validation",
6       "outcomes": [
7         {
8           "graderIndex": 0,
9           "kind": "output_match",
10           "passed": true,
11           "evidence": [
12             {
13               "index": 3,
14               "kind": "message"
15             },
16             {
17               "index": 7,
18               "kind": "tool_call"
19             }
20           ],
21           "reasoning": "Output matches expected login success message.",
22           "score": 1,
23           "severity": "gate",
24           "verdict": "pass"
25         },
26         {
27           "graderIndex": 1,
28           "kind": "response_time",
29           "passed": true,
30           "evidence": [
31             {
32               "index": 5,
33               "kind": "message"
34             }
35           ],
36           "reasoning": "Response time within acceptable threshold.",
37           "score": 1,
38           "severity": "soft",
39           "verdict": "pass"
40         }
41       ],
42       "outputExcerpt": "User logged in successfully.",
43       "passed": true
44     }
45   ],
46   "name": "User Authentication Suite",
47   "passed": true,
48   "passedCases": 1,
49   "runId": "a3f1c9e2-4b7d-4f8a-9c2e-1d2f3b4a5c6d",
50   "score": 1,
51   "suiteId": "auth-suite-2024",
52   "targetType": "flow",
53   "totalCases": 1
54 }

Run every enabled case of the suite against its target and grade the outputs with the suite graders. Suites within the synchronous case limit (50) run inline and return the full scored result; larger suites are queued as a durable run (202) that is graded when it completes — poll GET /eval/runs/{runId}/scores. Both paths persist per-case scores to the same run history.

Authentication

AuthorizationBearer

API key or Clerk session token

Path parameters

idstringRequired>=1 character

Request

This endpoint expects an object.

modeenumOptional

Force the execution path. Default: sync for suites within the synchronous case limit (50), batch above it.

Allowed values:

strictbooleanOptional

Strict grading: a soft grader miss fails its case like a gate miss.

Response

Suite ran synchronously — score + per-case grader outcomes

caseslist of objects

namestring

passedboolean

True when every case passed every grader.

passedCasesinteger

runIdstring or null

The persisted run id — query its scores via GET /eval/runs/{runId}/scores. Null for an ephemeral (virtual/inline) run.

scoredouble

Suite score, 0..1 (passed cases / total cases).

suiteIdstring or null

targetTypeenum

totalCasesinteger

Errors

400

Bad Request Error

401

Unauthorized Error

402

Payment Required Error

403

Forbidden Error

404

Not Found Error

500

Internal Server Error