Skip to navigation
Runtype APIEvals

Update eval suite

Update a suite’s Definition (name, description, graders) or state (default flag, baseline run pin). The target is immutable — create a new suite to evaluate a different flow/agent. Definition edits stamp provenance; a graders change invalidates the config-as-code content hash so the next ensure re-converges.

Authentication

AuthorizationBearer
API key or Clerk session token

Path parameters

idstringRequired>=1 character

Request

This endpoint expects an object.
baselineBatchExecutionIdstring or nullOptional

Pin a run of this suite as the comparison baseline; null clears the pin.

descriptionstring or nullOptional
graderslist of objectsOptional
identityDefaultsobject or nullOptional

Default identity (endUser / tenant) every case runs as unless the case sets its own; replaces the stored value. null clears it.

isDefaultbooleanOptional
namestringOptional>=1 character
recordedToolModeenumOptional

How captured (saved_from_run) cases replay. “next_step” captures the first tool intent and stops; “continue” serves recorded toolMocks and runs past the first action. null resets to the default (next_step).

Allowed values:
recordedToolUnmatchedPolicyenumOptional

For “continue”-mode replays, the unmatched-call policy: “fail” (divergence is the signal) or “stub” (canned success). null resets to the default (fail).

Allowed values:

Response

Suite updated
agentIdstring or null
baselineBatchExecutionIdstring or null
caseCountinteger
createdAtstring
descriptionstring or null
enabledCaseCountinteger
flowIdstring or null
graderslist of any

Ordered GraderConfig[] applied to every case.

idstring
identityDefaultsobject or null

Default identity (endUser / tenant) cases run as unless they set their own; null when unset.

isDefaultboolean
lastModifiedSourcestring or null

Provenance of the last Definition write. ‘sdk’ / ‘terraform’ mean the suite is managed in code (converged by ensure on deploy).

latestRunobject
namestring
recordedToolModeenum

How captured cases replay: “next_step” (default when null) or “continue”.

Allowed values:
recordedToolUnmatchedPolicyenum

Unmatched-call policy for “continue”-mode replays: “fail” (default when null) or “stub”.

Allowed values:
targetNamestring or null

Display name of the target flow/agent; null when the target row is gone.

targetTypeenum
Allowed values:
updatedAtstring
warningslist of objectsOptional

Non-blocking findings about this suite. AGENT_NOT_DEPLOYED means the target agent has no runnable live version, so a run that names no version selector refuses once the live default is on for the organization.

Errors

400
Bad Request Error
401
Unauthorized Error
403
Forbidden Error
404
Not Found Error
500
Internal Server Error