Create eval suite

POST

/v1/eval/suites

POST

/v1/eval/suites

$ curl -X POST https://api.runtype.com/v1/eval/suites \
>      -H "Authorization: <apiKey>" \
>      -H "Content-Type: application/json" \
>      -d '{
>   "name": "Customer Support Chatbot Eval Suite"
> }'

1 {
2   "agentId": "agent_9f8b7c6d5e4a3b2c1d0e",
3   "baselineBatchExecutionId": "batch_20240601_153000",
4   "caseCount": 3,
5   "cases": [
6     {
7       "createdAt": "2024-06-01T15:30:00Z",
8       "enabled": true,
9       "graders": [
10         null
11       ],
12       "id": "case_00123abc",
13       "name": "Greeting Message Test",
14       "notes": "Verify the bot greets the user appropriately.",
15       "origin": "manual",
16       "suiteId": "suite_456def789",
17       "updatedAt": "2024-06-01T15:30:00Z",
18       "input": null
19     }
20   ],
21   "createdAt": "2024-06-01T15:00:00Z",
22   "description": "Suite to evaluate the customer support chatbot's response quality and accuracy.",
23   "enabledCaseCount": 3,
24   "flowId": "flow_123abc456def",
25   "graders": [
26     null
27   ],
28   "id": "suite_456def789",
29   "isDefault": true,
30   "judgeAgreement": {
31     "agree": 42,
32     "total": 50
33   },
34   "lastModifiedSource": "dashboard",
35   "latestRun": {
36     "completedAt": "2024-06-02T10:00:00Z",
37     "passedCases": 40,
38     "runId": "run_789ghi012jkl",
39     "score": 0.85,
40     "startedAt": "2024-06-02T09:00:00Z",
41     "status": "completed",
42     "totalCases": 50
43   },
44   "name": "Customer Support Chatbot Eval Suite",
45   "recordedToolMode": "next_step",
46   "recordedToolUnmatchedPolicy": "fail",
47   "targetName": "Customer Support Chatbot",
48   "targetType": "flow",
49   "updatedAt": "2024-06-01T15:45:00Z"
50 }

Create an eval suite (named test cases + graders) attached to one flow or agent, optionally with initial cases. Suites created here are dashboard/API-managed; config-as-code suites converge via POST /eval/ensure instead.

Authentication

AuthorizationBearer

API key or Clerk session token

Request

This endpoint expects an object.

namestringRequired>=1 character

agentIdstringOptional>=1 character

caseslist of objectsOptional

descriptionstringOptional

flowIdstringOptional>=1 character

graderslist of objectsOptionalDefaults to []

Ordered grader configs applied to every case (suite-level graders).

isDefaultbooleanOptional

Make this the default eval for its target (clears the flag on any other suite for the same target).

recordedToolModeenumOptional

How captured (saved_from_run) cases replay. “next_step” (default) captures the first tool intent and stops; “continue” serves recorded toolMocks and runs past the first action.

Allowed values:

recordedToolUnmatchedPolicyenumOptional

For “continue”-mode replays, what to do when a tool call has no matching mock: “fail” (default, divergence is the signal) or “stub” (canned success so the run keeps going). A replay never reaches the real tool.

Allowed values:

Response

Suite created

agentIdstring or null

baselineBatchExecutionIdstring or null

caseCountinteger

caseslist of objects

createdAtstring

descriptionstring or null

enabledCaseCountinteger

flowIdstring or null

graderslist of any

Ordered GraderConfig[] applied to every case.

idstring

isDefaultboolean

judgeAgreementobject or null

The judge-trust corpus (E6 calibration): how often the AI grader agrees with your reviews, surfaced as a plain fraction. Null until at least one verdict has been reviewed (POST /eval/scores/{scoreId}/review).

lastModifiedSourcestring or null

Provenance of the last Definition write. ‘sdk’ / ‘terraform’ mean the suite is managed in code (converged by ensure on deploy).

latestRunobject

namestring

recordedToolModeenum

How captured cases replay: “next_step” (default when null) or “continue”.

recordedToolUnmatchedPolicyenum

Unmatched-call policy for “continue”-mode replays: “fail” (default when null) or “stub”.

targetNamestring or null

Display name of the target flow/agent; null when the target row is gone.

targetTypeenum

updatedAtstring

Errors

400

Bad Request Error

401

Unauthorized Error

403

Forbidden Error

404

Not Found Error

500

Internal Server Error