Get eval suite | Runtype Documentation

{
  "agentId": "string",
  "baselineBatchExecutionId": "string",
  "caseCount": 1,
  "cases": [
    {
      "createdAt": "string",
      "enabled": true,
      "graders": [
        null
      ],
      "id": "string",
      "name": "string",
      "notes": "string",
      "origin": "string",
      "suiteId": "string",
      "updatedAt": "string",
      "input": null
    }
  ],
  "createdAt": "string",
  "description": "string",
  "enabledCaseCount": 1,
  "flowId": "string",
  "graders": [
    null
  ],
  "id": "string",
  "identityDefaults": {
    "endUser": {
      "id": "string",
      "email": "string"
    },
    "tenant": {
      "id": "string"
    }
  },
  "isDefault": true,
  "judgeAgreement": {
    "agree": 1,
    "total": 1
  },
  "lastModifiedSource": "string",
  "latestRun": {
    "completedAt": "string",
    "passedCases": 1,
    "platformAbortedCases": 1,
    "runId": "string",
    "score": 1.1,
    "startedAt": "string",
    "status": "string",
    "totalCases": 1
  },
  "name": "string",
  "recordedToolMode": "next_step",
  "recordedToolUnmatchedPolicy": "fail",
  "targetName": "string",
  "targetType": "flow",
  "updatedAt": "string",
  "warnings": [
    {
      "code": "string",
      "message": "string",
      "agentId": "string"
    }
  ]
}

Get an eval suite including its cases and latest run + score.

Authentication

AuthorizationBearer

API key or Clerk session token

Path parameters

idstringRequired>=1 character

Response

Eval suite returned

agentIdstring or null

baselineBatchExecutionIdstring or null

caseCountinteger

caseslist of objects

createdAtstring

descriptionstring or null

enabledCaseCountinteger

flowIdstring or null

graderslist of any

Ordered GraderConfig[] applied to every case.

idstring

identityDefaultsobject or null

Default identity (endUser / tenant) cases run as unless they set their own; null when unset.

isDefaultboolean

judgeAgreementobject or null

The judge-trust corpus (E6 calibration): how often the AI grader agrees with your reviews, surfaced as a plain fraction. Null until at least one verdict has been reviewed (POST /eval/scores/{scoreId}/review).

lastModifiedSourcestring or null

Provenance of the last Definition write. ‘sdk’ / ‘terraform’ mean the suite is managed in code (converged by ensure on deploy).

latestRunobject

namestring

recordedToolModeenum

How captured cases replay: “next_step” (default when null) or “continue”.

Allowed values:

recordedToolUnmatchedPolicyenum

Unmatched-call policy for “continue”-mode replays: “fail” (default when null) or “stub”.

Allowed values:

targetNamestring or null

Display name of the target flow/agent; null when the target row is gone.

targetTypeenum

Allowed values:

updatedAtstring

warningslist of objectsOptional

Non-blocking findings about this suite. AGENT_NOT_DEPLOYED means the target agent has no runnable live version, so a run that names no version selector refuses once the live default is on for the organization.

Errors

401

Unauthorized Error

403

Forbidden Error

404

Not Found Error

500

Internal Server Error