Skip to navigation
Runtype APIEvals

Create eval suite

Create an eval suite (named test cases + graders) attached to one flow or agent, optionally with initial cases. Suites created here are dashboard/API-managed; config-as-code suites converge via POST /eval/ensure instead.

Authentication

AuthorizationBearer
API key or Clerk session token

Request

This endpoint expects an object.
namestringRequired>=1 character
agentIdstringOptional>=1 character
caseslist of objectsOptional
descriptionstringOptional
flowIdstringOptional>=1 character
graderslist of objectsOptionalDefaults to []

Ordered grader configs applied to every case (suite-level graders).

identityDefaultsobjectOptional

Default identity (endUser / tenant) every case runs as unless the case sets its own. Required (or per-case identity) when the agent sets config.tenancyStrategy. No credentials: the identity is asserted by the suite author.

isDefaultbooleanOptional

Make this the default eval for its target (clears the flag on any other suite for the same target).

recordedToolModeenumOptional

How captured (saved_from_run) cases replay. “next_step” (default) captures the first tool intent and stops; “continue” serves recorded toolMocks and runs past the first action.

Allowed values:
recordedToolUnmatchedPolicyenumOptional

For “continue”-mode replays, what to do when a tool call has no matching mock: “fail” (default, divergence is the signal) or “stub” (canned success so the run keeps going). A replay never reaches the real tool.

Allowed values:

Response

Suite created
agentIdstring or null
baselineBatchExecutionIdstring or null
caseCountinteger
caseslist of objects
createdAtstring
descriptionstring or null
enabledCaseCountinteger
flowIdstring or null
graderslist of any

Ordered GraderConfig[] applied to every case.

idstring
identityDefaultsobject or null

Default identity (endUser / tenant) cases run as unless they set their own; null when unset.

isDefaultboolean
judgeAgreementobject or null

The judge-trust corpus (E6 calibration): how often the AI grader agrees with your reviews, surfaced as a plain fraction. Null until at least one verdict has been reviewed (POST /eval/scores/{scoreId}/review).

lastModifiedSourcestring or null

Provenance of the last Definition write. ‘sdk’ / ‘terraform’ mean the suite is managed in code (converged by ensure on deploy).

latestRunobject
namestring
recordedToolModeenum

How captured cases replay: “next_step” (default when null) or “continue”.

Allowed values:
recordedToolUnmatchedPolicyenum

Unmatched-call policy for “continue”-mode replays: “fail” (default when null) or “stub”.

Allowed values:
targetNamestring or null

Display name of the target flow/agent; null when the target row is gone.

targetTypeenum
Allowed values:
updatedAtstring
warningslist of objectsOptional

Non-blocking findings about this suite. AGENT_NOT_DEPLOYED means the target agent has no runnable live version, so a run that names no version selector refuses once the live default is on for the organization.

Errors

400
Bad Request Error
401
Unauthorized Error
403
Forbidden Error
404
Not Found Error
500
Internal Server Error