Get eval suite

GET

/v1/eval/suites/:id

GET

/v1/eval/suites/:id

$ curl https://api.runtype.com/v1/eval/suites/id \
>      -H "Authorization: <apiKey>" \
>      -H "Content-Type: application/json"

1 {
2   "agentId": "agent_9f8b7c6d",
3   "baselineBatchExecutionId": "batch_exec_20240601",
4   "caseCount": 3,
5   "cases": [
6     {
7       "createdAt": "2024-05-20T10:15:30Z",
8       "enabled": true,
9       "graders": null,
10       "id": "case_123abc",
11       "name": "Weather Query Case",
12       "notes": "Tests basic weather query response",
13       "origin": "imported",
14       "suiteId": "suite_456def",
15       "updatedAt": "2024-05-22T08:45:00Z",
16       "input": {
17         "variables": {
18           "userMessage": "What is the weather like today?"
19         }
20       }
21     }
22   ],
23   "createdAt": "2024-05-15T09:00:00Z",
24   "description": "Suite to evaluate chatbot responses for common queries",
25   "enabledCaseCount": 3,
26   "flowId": "flow_789ghi",
27   "graders": [],
28   "id": "suite_456def",
29   "isDefault": false,
30   "judgeAgreement": {
31     "agree": 15,
32     "total": 20
33   },
34   "lastModifiedSource": "sdk",
35   "latestRun": {
36     "completedAt": "2024-06-01T12:00:00Z",
37     "passedCases": 14,
38     "runId": "run_20240601_01",
39     "score": 0.93,
40     "startedAt": "2024-06-01T11:00:00Z",
41     "status": "completed",
42     "totalCases": 15
43   },
44   "name": "Chatbot Response Evaluation",
45   "recordedToolMode": "next_step",
46   "recordedToolUnmatchedPolicy": "fail",
47   "targetName": "Customer Support Flow",
48   "targetType": "flow",
49   "updatedAt": "2024-06-01T12:05:00Z"
50 }

Get an eval suite including its cases and latest run + score.

Authentication

AuthorizationBearer

API key or Clerk session token

Path parameters

idstringRequired>=1 character

Response

Eval suite returned

agentIdstring or null

baselineBatchExecutionIdstring or null

caseCountinteger

caseslist of objects

createdAtstring

descriptionstring or null

enabledCaseCountinteger

flowIdstring or null

graderslist of any

Ordered GraderConfig[] applied to every case.

idstring

isDefaultboolean

judgeAgreementobject or null

The judge-trust corpus (E6 calibration): how often the AI grader agrees with your reviews, surfaced as a plain fraction. Null until at least one verdict has been reviewed (POST /eval/scores/{scoreId}/review).

lastModifiedSourcestring or null

Provenance of the last Definition write. ‘sdk’ / ‘terraform’ mean the suite is managed in code (converged by ensure on deploy).

latestRunobject

namestring

recordedToolModeenum

How captured cases replay: “next_step” (default when null) or “continue”.

recordedToolUnmatchedPolicyenum

Unmatched-call policy for “continue”-mode replays: “fail” (default when null) or “stub”.

targetNamestring or null

Display name of the target flow/agent; null when the target row is gone.

targetTypeenum

updatedAtstring

Errors

401

Unauthorized Error

403

Forbidden Error

404

Not Found Error

500

Internal Server Error