> For clean Markdown of any page, append .md to the page URL.
> For a complete documentation index, see https://docs.runtype.com/llms.txt.
> For AI client integration (Claude Code, Cursor, etc.), connect to the MCP server at https://docs.runtype.com/_mcp/server.

# Stream eval execution

POST https://api.runtype.com/v1/eval/stream
Content-Type: application/json

Execute virtual eval with record/messages input and stream results via multiplexed SSE. All eval configs execute simultaneously in parallel.

Reference: https://docs.runtype.com/api-reference/runtype-api/evals/stream-eval-execution

## Authentication

- `Authorization` header (bearer token, required) — API key or Clerk session token

## Servers

- `https://api.runtype.com` (Production, default)
- `https://api.runtype-staging.com` (Staging)

## Request

### Body (application/json)

This endpoint expects an object.

- `evalConfigs` (list of EvalConfigRequest, optional)

## Response

### 200

SSE stream of eval results

- Streaming response of `ExecutionStreamEvent`.
- ExecutionStreamEvent0
- ExecutionStreamEvent1
- ExecutionStreamEvent2
- ExecutionStreamEvent3
- ExecutionStreamEvent4
- ExecutionStreamEvent5
- ExecutionStreamEvent6
- ExecutionStreamEvent7
- ExecutionStreamEvent8
- ExecutionStreamEvent9
- ExecutionStreamEvent10
- ExecutionStreamEvent11
- ExecutionStreamEvent12
- ExecutionStreamEvent13
- ExecutionStreamEvent14
- ExecutionStreamEvent15
- ExecutionStreamEvent16
- ExecutionStreamEvent17
- ExecutionStreamEvent18
- ExecutionStreamEvent19
- ExecutionStreamEvent20
- ExecutionStreamEvent21
- ExecutionStreamEvent22
- ExecutionStreamEvent23
- ExecutionStreamEvent24
- ExecutionStreamEvent25
- ExecutionStreamEvent26
- ExecutionStreamEvent27
- ExecutionStreamEvent28
- ExecutionStreamEvent29
- ExecutionStreamEvent30
- ExecutionStreamEvent31
- ExecutionStreamEvent32
- ExecutionStreamEvent33
- ExecutionStreamEvent34
- ExecutionStreamEvent35
- ExecutionStreamEvent36
- ExecutionStreamEvent37

## Errors

### 400 Bad Request Error

Invalid request

- `error` (string, required)
- `code` (string, optional) — Stable error code. `insufficient_scope` marks an API key that lacks a scope this request needs.
- `details` (list of ErrorDetailsItems, optional)
- `requiredScopes` (list of string, optional) — On `insufficient_scope`: every API key scope this request needs.

### 401 Unauthorized Error

Unauthorized

- `error` (string, required)
- `code` (string, optional) — Stable error code. `insufficient_scope` marks an API key that lacks a scope this request needs.
- `details` (list of ErrorDetailsItems, optional)
- `requiredScopes` (list of string, optional) — On `insufficient_scope`: every API key scope this request needs.

### 402 Payment Required Error

Limit exceeded

- `code` (string, required) — Stable error code. DAILY_EVAL_LIMIT: the daily eval count of the plan is used up; retry after `stats.resetsAt`. SPEND_LIMIT_EXCEEDED: the monthly spend cap is reached. CLAUDE_MANAGED_AGENTS_DENIED: the account cannot evaluate a Claude Managed agent.
- `error` (string, required)
- `message` (string, required) — Human-readable reason, including the limit and when it resets.
- `details` (list of EvalQuotaExceededErrorDetailsItems, optional)
- `requiredScopes` (list of string, optional) — On `insufficient_scope`: every API key scope this request needs.
- `stats` (EvalQuotaExceededErrorStats, optional) — Daily eval usage at the moment of a DAILY_EVAL_LIMIT or SPEND_LIMIT_EXCEEDED refusal.

### 403 Forbidden Error

Forbidden

- `error` (string, required)
- `code` (string, optional) — Stable error code. `insufficient_scope` marks an API key that lacks a scope this request needs.
- `details` (list of ErrorDetailsItems, optional)
- `requiredScopes` (list of string, optional) — On `insufficient_scope`: every API key scope this request needs.

### 500 Internal Server Error

Internal server error

- `code` (string, required) — Stable error code: `INTERNAL_ERROR` unless the route documents its own.
- `error` (string, required)
- `details` (list of InternalErrorDetailsItems, optional)
- `requiredScopes` (list of string, optional) — On `insufficient_scope`: every API key scope this request needs.
- `message` (string, optional) — Plain-language guidance: retry, then contact support with the `requestId`.
- `requestId` (string, optional) — The `X-Runtype-Request-ID` of the failed request, for support.

### 503 Service Unavailable Error

Database temporarily unavailable; nothing was read or written. Retry after the `Retry-After` interval (seconds).

- `code` (enum, required) — Stable error code. `insufficient_scope` marks an API key that lacks a scope this request needs.
  - Allowed values: `DATABASE_UNAVAILABLE`
- `error` (string, required)
- `details` (list of DatabaseUnavailableErrorDetailsItems, optional)
- `requiredScopes` (list of string, optional) — On `insufficient_scope`: every API key scope this request needs.
- `requestId` (string, optional) — The `X-Runtype-Request-ID` of the failed request, for support.

## Types

### EvalConfigRequest

- `evalConfigId` (string, optional)
- `evalName` (string, optional)
- `overrides` (EvalOverrides, optional) — Canonical eval override field. Replaces `stepOverrides` (and its `"*"` key), `claudeManagedOverride` and `advisorOverride`; setting both spellings of one channel is a 400.

### ExecutionStreamEvent0

- `executionId` (string, required)
- `kind` (enum, required)
  - Allowed values: `agent`, `flow`
- `seq` (double, required)
- `startedAt` (string, required)
- `type` (enum, required)
  - Allowed values: `execution_start`
- `agentId` (string, optional)
- `agentName` (string, optional)
- `config` (map from string to any, optional)
- `flowId` (string, optional)
- `flowName` (string, optional)
- `maxTurns` (double, optional)
- `resumed` (boolean, optional)
- `source` (string, optional)
- `totalSteps` (double, optional)

### ExecutionStreamEvent1

- `executionId` (string, required)
- `kind` (enum, required)
  - Allowed values: `agent`, `flow`
- `seq` (double, required)
- `success` (boolean, required)
- `type` (enum, required)
  - Allowed values: `execution_complete`
- `claudeManagedAgentId` (string, optional)
- `completedAt` (string, optional)
- `durationMs` (double, optional)
- `externalAgent` (ExecutionStreamEventOneOf1ExternalAgent, optional)
- `failedSteps` (double, optional)
- `finalOutput` (string, optional)
- `iterations` (double, optional)
- `outcome` (enum, optional) — How the execution ended: `completed`, `error`, or `aborted` (cancelled, interrupted, or superseded by the caller or host). Absent on frames from older producers: treat `execution_error` as `error` and `execution_complete` as `completed`, or `error` when `success` is false.
  - Allowed values: `completed`, `error`, `aborted`
- `stepErrorCount` (double, optional)
- `stopReason` (string, optional)
- `successfulSteps` (double, optional)
- `totalCost` (double, optional)
- `totalSteps` (double, optional)
- `totalTokens` (ExecutionStreamEventOneOf1TotalTokens, optional)
- `totalTokensUsed` (double, optional)

### ExecutionStreamEvent2

- `error` (ExecutionStreamEventOneOf2Error, required)
- `executionId` (string, required)
- `kind` (enum, required)
  - Allowed values: `agent`, `flow`
- `seq` (double, required)
- `type` (enum, required)
  - Allowed values: `execution_error`
- `blockReason` (string, optional)
- `code` (string, optional)
- `completedAt` (string, optional)
- `finalOutput` (string, optional)
- `outcome` (enum, optional) — How the execution ended: `completed`, `error`, or `aborted` (cancelled, interrupted, or superseded by the caller or host). Absent on frames from older producers: treat `execution_error` as `error` and `execution_complete` as `completed`, or `error` when `success` is false.
  - Allowed values: `completed`, `error`, `aborted`
- `stopReason` (string, optional)
- `upgradeUrl` (string, optional)

### ExecutionStreamEvent3

- `executionId` (string, required)
- `id` (string, required)
- `role` (enum, required)
  - Allowed values: `user`, `assistant`, `system`
- `seq` (double, required)
- `type` (enum, required)
  - Allowed values: `turn_start`
- `iteration` (double, optional)
- `subagent` (ExecutionStreamEventOneOf3Subagent, optional)
- `turnIndex` (double, optional)

### ExecutionStreamEvent4

- `executionId` (string, required)
- `id` (string, required)
- `role` (enum, required)
  - Allowed values: `user`, `assistant`, `system`
- `seq` (double, required)
- `type` (enum, required)
  - Allowed values: `turn_complete`
- `attributes` (map from string to ExecutionStreamEventOneOf4Attributes, optional)
- `completedAt` (string, optional)
- `content` (string, optional)
- `cost` (double, optional)
- `estimatedContextBreakdown` (ExecutionStreamEventOneOf4EstimatedContextBreakdown, optional)
- `fallback` (ExecutionStreamEventOneOf4Fallback, optional)
- `iteration` (double, optional)
- `stopReason` (enum, optional)
  - Allowed values: `end_turn`, `max_tool_calls`, `length`, `content_filter`, `error`, `unknown`
- `subagent` (ExecutionStreamEventOneOf4Subagent, optional)
- `tokens` (ExecutionStreamEventOneOf4Tokens, optional)

### ExecutionStreamEvent5

- `executionId` (string, required)
- `id` (string, required)
- `ordinal` (integer, required)
- `seq` (double, required)
- `type` (enum, required)
  - Allowed values: `model_call_start`
- `iteration` (double, optional)
- `model` (string, optional)
- `provider` (string, optional)
- `startedAt` (string, optional)
- `stepId` (string, optional)
- `subagent` (ExecutionStreamEventOneOf5Subagent, optional)
- `turnId` (string, optional)

### ExecutionStreamEvent6

- `executionId` (string, required)
- `id` (string, required)
- `ordinal` (integer, required)
- `seq` (double, required)
- `type` (enum, required)
  - Allowed values: `model_call_complete`
- `attributes` (map from string to ExecutionStreamEventOneOf6Attributes, optional)
- `completedAt` (string, optional)
- `cost` (double, optional)
- `durationMs` (double, optional)
- `finishReason` (string, optional)
- `iteration` (double, optional)
- `model` (string, optional)
- `provider` (string, optional)
- `responseModel` (string, optional)
- `subagent` (ExecutionStreamEventOneOf6Subagent, optional)
- `success` (boolean, optional)
- `tokens` (ExecutionStreamEventOneOf6Tokens, optional)
- `turnId` (string, optional)

### ExecutionStreamEvent7

- `executionId` (string, required)
- `id` (string, required)
- `seq` (double, required)
- `type` (enum, required)
  - Allowed values: `step_start`
- `index` (double, optional)
- `name` (string, optional)
- `outputVariable` (string, optional)
- `startedAt` (string, optional)
- `stepType` (string, optional)
- `subagent` (ExecutionStreamEventOneOf7Subagent, optional)
- `totalSteps` (double, optional)

### ExecutionStreamEvent8

- `executionId` (string, required)
- `id` (string, required)
- `seq` (double, required)
- `type` (enum, required)
  - Allowed values: `step_complete`
- `completedAt` (string, optional)
- `cost` (double, optional)
- `durationMs` (double, optional)
- `error` (string, optional)
- `fallback` (ExecutionStreamEventOneOf8Fallback, optional)
- `name` (string, optional)
- `result` (any, optional)
- `stepType` (string, optional)
- `stopReason` (enum, optional)
  - Allowed values: `end_turn`, `max_tool_calls`, `length`, `content_filter`, `error`, `unknown`
- `subagent` (ExecutionStreamEventOneOf8Subagent, optional)
- `success` (boolean, optional)
- `tokensUsed` (double, optional)
- `unresolvedVariables` (list of string, optional)

### ExecutionStreamEvent9

- `executionId` (string, required)
- `id` (string, required)
- `seq` (double, required)
- `type` (enum, required)
  - Allowed values: `step_skip`
- `index` (double, optional)
- `name` (string, optional)
- `skippedAt` (string, optional)
- `stepType` (string, optional)
- `totalSteps` (double, optional)
- `when` (string, optional)

### ExecutionStreamEvent10

- `executionId` (string, required)
- `id` (string, required)
- `seq` (double, required)
- `type` (enum, required)
  - Allowed values: `text_start`
- `parentToolCallId` (string, optional)
- `role` (enum, optional)
  - Allowed values: `user`, `assistant`, `system`
- `stepId` (string, optional)
- `subagent` (ExecutionStreamEventOneOf10Subagent, optional)
- `turnId` (string, optional)

### ExecutionStreamEvent11

- `delta` (string, required)
- `executionId` (string, required)
- `id` (string, required)
- `seq` (double, required)
- `type` (enum, required)
  - Allowed values: `text_delta`
- `subagent` (ExecutionStreamEventOneOf11Subagent, optional)

### ExecutionStreamEvent12

- `executionId` (string, required)
- `id` (string, required)
- `seq` (double, required)
- `type` (enum, required)
  - Allowed values: `text_complete`
- `subagent` (ExecutionStreamEventOneOf12Subagent, optional)
- `text` (string, optional)

### ExecutionStreamEvent13

- `executionId` (string, required)
- `id` (string, required)
- `seq` (double, required)
- `type` (enum, required)
  - Allowed values: `reasoning_start`
- `parentToolCallId` (string, optional)
- `scope` (enum, optional)
  - Allowed values: `turn`, `loop`
- `subagent` (ExecutionStreamEventOneOf13Subagent, optional)

### ExecutionStreamEvent14

- `delta` (string, required)
- `executionId` (string, required)
- `id` (string, required)
- `seq` (double, required)
- `type` (enum, required)
  - Allowed values: `reasoning_delta`
- `subagent` (ExecutionStreamEventOneOf14Subagent, optional)

### ExecutionStreamEvent15

- `executionId` (string, required)
- `id` (string, required)
- `seq` (double, required)
- `type` (enum, required)
  - Allowed values: `reasoning_complete`
- `scope` (enum, optional)
  - Allowed values: `turn`, `loop`
- `subagent` (ExecutionStreamEventOneOf15Subagent, optional)
- `text` (string, optional)

### ExecutionStreamEvent16

- `executionId` (string, required)
- `id` (string, required)
- `mediaType` (string, required)
- `seq` (double, required)
- `type` (enum, required)
  - Allowed values: `media_start`
- `role` (enum, optional)
  - Allowed values: `user`, `assistant`, `system`
- `toolCallId` (string, optional)

### ExecutionStreamEvent17

- `delta` (string, required)
- `executionId` (string, required)
- `id` (string, required)
- `seq` (double, required)
- `type` (enum, required)
  - Allowed values: `media_delta`

### ExecutionStreamEvent18

- `executionId` (string, required)
- `id` (string, required)
- `seq` (double, required)
- `type` (enum, required)
  - Allowed values: `media_complete`
- `data` (string, optional)
- `mediaType` (string, optional)
- `toolCallId` (string, optional)
- `url` (string, optional)

### ExecutionStreamEvent19

- `artifactType` (enum, required)
  - Allowed values: `markdown`, `component`
- `id` (string, required)
- `type` (enum, required)
  - Allowed values: `artifact_start`
- `component` (string, optional)
- `executionId` (string, optional)
- `file` (ExecutionStreamEventOneOf19File, optional)
- `seq` (double, optional)
- `title` (string, optional)

### ExecutionStreamEvent20

- `delta` (string, required)
- `id` (string, required)
- `type` (enum, required)
  - Allowed values: `artifact_delta`
- `executionId` (string, optional)
- `seq` (double, optional)

### ExecutionStreamEvent21

- `component` (string, required)
- `id` (string, required)
- `props` (map from string to any, required)
- `type` (enum, required)
  - Allowed values: `artifact_update`
- `executionId` (string, optional)
- `seq` (double, optional)

### ExecutionStreamEvent22

- `id` (string, required)
- `type` (enum, required)
  - Allowed values: `artifact_complete`
- `executionId` (string, optional)
- `seq` (double, optional)

### ExecutionStreamEvent23

- `executionId` (string, required)
- `seq` (double, required)
- `type` (enum, required)
  - Allowed values: `source`
- `id` (string, optional)
- `sourceType` (string, optional)
- `title` (string, optional)
- `url` (string, optional)

### ExecutionStreamEvent24

- `executionId` (string, required)
- `seq` (double, required)
- `state` (map from string to any, required)
- `type` (enum, required)
  - Allowed values: `state_snapshot`
- `iteration` (double, optional)

### ExecutionStreamEvent25

- `executionId` (string, required)
- `patch` (list of ExecutionStreamEventOneOf25PatchItems, required)
- `seq` (double, required)
- `type` (enum, required)
  - Allowed values: `state_delta`
- `iteration` (double, optional)

### ExecutionStreamEvent26

- `executionId` (string, required)
- `seq` (double, required)
- `toolCallId` (string, required)
- `toolName` (string, required)
- `toolType` (string, required)
- `type` (enum, required)
  - Allowed values: `tool_start`
- `annotations` (ExecutionStreamEventOneOf26Annotations, optional)
- `hiddenParameterNames` (list of string, optional)
- `iteration` (double, optional)
- `origin` (enum, optional)
  - Allowed values: `webmcp`, `sdk`
- `pageOrigin` (string, optional)
- `parameters` (map from string to any, optional)
- `startedAt` (string, optional)
- `stepId` (string, optional)
- `subagent` (ExecutionStreamEventOneOf26Subagent, optional)

### ExecutionStreamEvent27

- `delta` (string, required)
- `executionId` (string, required)
- `seq` (double, required)
- `toolCallId` (string, required)
- `type` (enum, required)
  - Allowed values: `tool_input_delta`
- `subagent` (ExecutionStreamEventOneOf27Subagent, optional)

### ExecutionStreamEvent28

- `executionId` (string, required)
- `parameters` (map from string to any, required)
- `seq` (double, required)
- `toolCallId` (string, required)
- `type` (enum, required)
  - Allowed values: `tool_input_complete`
- `hiddenParameterNames` (list of string, optional)
- `subagent` (ExecutionStreamEventOneOf28Subagent, optional)
- `toolName` (string, optional)

### ExecutionStreamEvent29

- `delta` (string, required)
- `executionId` (string, required)
- `seq` (double, required)
- `toolCallId` (string, required)
- `type` (enum, required)
  - Allowed values: `tool_output_delta`

### ExecutionStreamEvent30

- `executionId` (string, required)
- `seq` (double, required)
- `success` (boolean, required)
- `toolCallId` (string, required)
- `type` (enum, required)
  - Allowed values: `tool_complete`
- `error` (string, optional)
- `executionTime` (double, optional)
- `iteration` (double, optional)
- `result` (any, optional)
- `stepId` (string, optional)
- `subagent` (ExecutionStreamEventOneOf30Subagent, optional)
- `toolCost` (double, optional)
- `toolName` (string, optional)

### ExecutionStreamEvent31

- `approvalId` (string, required)
- `executionId` (string, required)
- `seq` (double, required)
- `toolName` (string, required)
- `type` (enum, required)
  - Allowed values: `approval_start`
- `description` (string, optional)
- `externalAgent` (ExecutionStreamEventOneOf31ExternalAgent, optional)
- `iteration` (double, optional)
- `parameters` (map from string to any, optional)
- `reason` (string, optional)
- `startedAt` (string, optional)
- `subagent` (ExecutionStreamEventOneOf31Subagent, optional)
- `timeout` (double, optional)
- `toolCallId` (string, optional)
- `toolType` (string, optional)

### ExecutionStreamEvent32

- `approvalId` (string, required)
- `decision` (enum, required)
  - Allowed values: `approved`, `denied`, `timeout`
- `executionId` (string, required)
- `seq` (double, required)
- `type` (enum, required)
  - Allowed values: `approval_complete`
- `completedAt` (string, optional)
- `resolvedBy` (enum, optional)
  - Allowed values: `user`, `system`

### ExecutionStreamEvent33

- `executionId` (string, required)
- `seq` (double, required)
- `type` (enum, required)
  - Allowed values: `await`
- `annotations` (ExecutionStreamEventOneOf33Annotations, optional)
- `awaitReason` (string, optional)
- `awaitedAt` (string, optional)
- `crawlId` (string, optional)
- `elicitation` (ExecutionStreamEventOneOf33Elicitation, optional)
- `externalAgent` (ExecutionStreamEventOneOf33ExternalAgent, optional)
- `origin` (enum, optional)
  - Allowed values: `webmcp`, `sdk`
- `pageOrigin` (string, optional)
- `parameters` (map from string to any, optional)
- `stepId` (string, optional)
- `toolCallId` (string, optional)
- `toolId` (string, optional)
- `toolName` (string, optional)

### ExecutionStreamEvent34

- `error` (ExecutionStreamEventOneOf34Error, required)
- `executionId` (string, required)
- `seq` (double, required)
- `type` (enum, required)
  - Allowed values: `error`
- `recoverable` (boolean, optional)

### ExecutionStreamEvent35

- `epoch` (integer, required)
- `executionId` (string, required)
- `kind` (enum, required)
  - Allowed values: `compaction`
- `phase` (enum, required)
  - Allowed values: `start`, `complete`
- `reason` (enum, required)
  - Allowed values: `threshold`, `overflow`, `manual`, `surface_policy`, `provider_mismatch`
- `seq` (double, required)
- `strategy` (enum, required)
  - Allowed values: `provider_native`, `summary`
- `tier` (integer, required)
- `type` (enum, required)
  - Allowed values: `context_notice`
- `outcome` (enum, optional)
  - Allowed values: `ok`, `failed`, `gap`, `skipped`
- `tokensAfter` (double, optional)
- `tokensBefore` (double, optional)

### ExecutionStreamEvent36

- `executionId` (string, required)
- `seq` (double, required)
- `timestamp` (string, required)
- `type` (enum, required)
  - Allowed values: `ping`

### ExecutionStreamEvent37

- `name` (string, required)
- `type` (enum, required)
  - Allowed values: `custom`
- `executionId` (string, optional)
- `seq` (double, optional)
- `value` (any, optional)

### ErrorDetailsItems

- `message` (string, required)
- `code` (string, optional) — Stable Runtype validation code (e.g. MISSING_REQUIRED_FIELD, INVALID_ENUM).
- `path` (string, optional) — Dotted JSON path to the offending field.
- `suggestion` (string, optional) — Actionable hint (allowed values, accepted range, …).

### EvalQuotaExceededErrorDetailsItems

- `message` (string, required)
- `code` (string, optional) — Stable Runtype validation code (e.g. MISSING_REQUIRED_FIELD, INVALID_ENUM).
- `path` (string, optional) — Dotted JSON path to the offending field.
- `suggestion` (string, optional) — Actionable hint (allowed values, accepted range, …).

### EvalQuotaExceededErrorStats

Daily eval usage at the moment of a DAILY_EVAL_LIMIT or SPEND_LIMIT_EXCEEDED refusal.

- `evalLimit` (integer, required) — Daily eval limit; -1 means unlimited.
- `evalsUsed` (integer, required)
- `isAtLimit` (boolean, required)
- `remaining` (double, required, nullable)
- `resetsAt` (string, required) — ISO 8601 instant the daily count resets (midnight UTC).

### InternalErrorDetailsItems

- `message` (string, required)
- `code` (string, optional) — Stable Runtype validation code (e.g. MISSING_REQUIRED_FIELD, INVALID_ENUM).
- `path` (string, optional) — Dotted JSON path to the offending field.
- `suggestion` (string, optional) — Actionable hint (allowed values, accepted range, …).

### DatabaseUnavailableErrorDetailsItems

- `message` (string, required)
- `code` (string, optional) — Stable Runtype validation code (e.g. MISSING_REQUIRED_FIELD, INVALID_ENUM).
- `path` (string, optional) — Dotted JSON path to the offending field.
- `suggestion` (string, optional) — Actionable hint (allowed values, accepted range, …).

### EvalOverrides

Canonical eval override field. Replaces `stepOverrides` (and its `"*"` key), `claudeManagedOverride` and `advisorOverride`; setting both spellings of one channel is a 400.

- `advisor` (EvalOverridesAdvisor, optional, nullable) — Advisor override for agent capabilities. `null` disables the advisor; omitting it leaves the target’s own config alone.
- `agent` (EvalStepOverrideValues, optional) — Applies to every prompt step of the agent under test. Valid only on an agent target; sending it on a flow target is a 400.
- `claudeManaged` (EvalOverridesClaudeManaged, optional) — Claude Managed variant override. Valid only on a claude_managed agent target; rejected on a standard agent rather than silently ignored.
- `flow` (EvalOverridesFlow, optional) — Applies to every prompt step of the flow under test. Valid only on a flow target; sending it on an agent target is a 400.
- `steps` (map from string to EvalOverridesSteps, optional) — Per-step overrides keyed by step id. Beats agent/flow field by field. A key naming no step of the target is a 400 rather than a silently-scored no-op.
- `subagentDefaults` (EvalOverridesSubagentDefaults, optional) — Applies to every reachable sub-agent with no `subagents` entry of its own. Does not apply to a dynamically-spawned child, which keeps the model its spawn resolved.
- `subagents` (map from string to EvalOverridesSubagents, optional) — Per-sub-agent overrides keyed by agentId. The key is an identity, not a position: an entry applies wherever that agent runs, at any depth. A key naming an agent this target cannot reach is a 400.

### ExecutionStreamEventOneOf1ExternalAgent

- `contextId` (string, optional)
- `taskId` (string, optional)

### ExecutionStreamEventOneOf1TotalTokens

- `input` (double, required)
- `output` (double, required)

### ExecutionStreamEventOneOf2Error

- string
- ExecutionStreamEventOneOf2Error1

### ExecutionStreamEventOneOf3Subagent

- `toolName` (string, required)
- `agentName` (string, optional)
- `parentToolCallId` (string, optional)

### ExecutionStreamEventOneOf4Attributes

- string
- double
- boolean

### ExecutionStreamEventOneOf4EstimatedContextBreakdown

- `categories` (map from string to ExecutionStreamEventOneOf4EstimatedContextBreakdownCategories, required)
- `estimatedTotalTokens` (double, required)
- `contextWindowSize` (double, optional)
- `estimatedUtilizationPercent` (double, optional)

### ExecutionStreamEventOneOf4Fallback

- `attempts` (list of ExecutionStreamEventOneOf4FallbackAttemptsItems, required)
- `exhausted` (boolean, required)
- `used` (boolean, required)
- `model` (string, optional)
- `reason` (string, optional, nullable)

### ExecutionStreamEventOneOf4Subagent

- `toolName` (string, required)
- `agentName` (string, optional)
- `parentToolCallId` (string, optional)

### ExecutionStreamEventOneOf4Tokens

- `input` (double, required)
- `output` (double, required)

### ExecutionStreamEventOneOf5Subagent

- `toolName` (string, required)
- `agentName` (string, optional)
- `parentToolCallId` (string, optional)

### ExecutionStreamEventOneOf6Attributes

- string
- double
- boolean

### ExecutionStreamEventOneOf6Subagent

- `toolName` (string, required)
- `agentName` (string, optional)
- `parentToolCallId` (string, optional)

### ExecutionStreamEventOneOf6Tokens

- `input` (double, required)
- `output` (double, required)
- `cacheRead` (double, optional)
- `cacheWrite` (double, optional)
- `reasoning` (double, optional)

### ExecutionStreamEventOneOf7Subagent

- `toolName` (string, required)
- `agentName` (string, optional)
- `parentToolCallId` (string, optional)

### ExecutionStreamEventOneOf8Fallback

- `attempts` (list of ExecutionStreamEventOneOf8FallbackAttemptsItems, required)
- `exhausted` (boolean, required)
- `used` (boolean, required)
- `model` (string, optional)
- `reason` (string, optional, nullable)

### ExecutionStreamEventOneOf8Subagent

- `toolName` (string, required)
- `agentName` (string, optional)
- `parentToolCallId` (string, optional)

### ExecutionStreamEventOneOf10Subagent

- `toolName` (string, required)
- `agentName` (string, optional)
- `parentToolCallId` (string, optional)

### ExecutionStreamEventOneOf11Subagent

- `toolName` (string, required)
- `agentName` (string, optional)
- `parentToolCallId` (string, optional)

### ExecutionStreamEventOneOf12Subagent

- `toolName` (string, required)
- `agentName` (string, optional)
- `parentToolCallId` (string, optional)

### ExecutionStreamEventOneOf13Subagent

- `toolName` (string, required)
- `agentName` (string, optional)
- `parentToolCallId` (string, optional)

### ExecutionStreamEventOneOf14Subagent

- `toolName` (string, required)
- `agentName` (string, optional)
- `parentToolCallId` (string, optional)

### ExecutionStreamEventOneOf15Subagent

- `toolName` (string, required)
- `agentName` (string, optional)
- `parentToolCallId` (string, optional)

### ExecutionStreamEventOneOf19File

- `mimeType` (string, required)
- `path` (string, required)
- `language` (string, optional)

### ExecutionStreamEventOneOf25PatchItems

- `op` (enum, required)
  - Allowed values: `add`, `remove`, `replace`, `move`, `copy`, `test`
- `path` (string, required)
- `from` (string, optional)
- `value` (any, optional)

### ExecutionStreamEventOneOf26Annotations

- `consequentialHint` (boolean, optional)
- `debugging` (boolean, optional)
- `destructiveHint` (boolean, optional)
- `idempotentHint` (boolean, optional)
- `openWorldHint` (boolean, optional)
- `readOnlyHint` (boolean, optional)
- `untrustedContentHint` (boolean, optional)

### ExecutionStreamEventOneOf26Subagent

- `toolName` (string, required)
- `agentName` (string, optional)
- `parentToolCallId` (string, optional)

### ExecutionStreamEventOneOf27Subagent

- `toolName` (string, required)
- `agentName` (string, optional)
- `parentToolCallId` (string, optional)

### ExecutionStreamEventOneOf28Subagent

- `toolName` (string, required)
- `agentName` (string, optional)
- `parentToolCallId` (string, optional)

### ExecutionStreamEventOneOf30Subagent

- `toolName` (string, required)
- `agentName` (string, optional)
- `parentToolCallId` (string, optional)

### ExecutionStreamEventOneOf31ExternalAgent

- `contextId` (string, optional)
- `taskId` (string, optional)

### ExecutionStreamEventOneOf31Subagent

- `toolName` (string, required)
- `agentName` (string, optional)
- `parentToolCallId` (string, optional)

### ExecutionStreamEventOneOf33Annotations

- `consequentialHint` (boolean, optional)
- `debugging` (boolean, optional)
- `destructiveHint` (boolean, optional)
- `idempotentHint` (boolean, optional)
- `openWorldHint` (boolean, optional)
- `readOnlyHint` (boolean, optional)
- `untrustedContentHint` (boolean, optional)

### ExecutionStreamEventOneOf33Elicitation

- `message` (string, required)
- `mode` (enum, required)
  - Allowed values: `form`, `url`
- `pauseCount` (double, optional)
- `requestedSchema` (map from string to any, optional)
- `requests` (list of ExecutionStreamEventOneOf33ElicitationRequestsItems, optional)
- `serverName` (string, optional)
- `url` (string, optional)

### ExecutionStreamEventOneOf33ExternalAgent

- `contextId` (string, optional)
- `taskId` (string, optional)

### ExecutionStreamEventOneOf34Error

- string
- ExecutionStreamEventOneOf34Error1

### EvalOverridesAdvisor

Advisor override for agent capabilities. `null` disables the advisor; omitting it leaves the target’s own config alone.

- `model` (string, required)
- `systemPrompt` (string, optional)

### EvalStepOverrideValues

Applies to every prompt step of the agent under test. Valid only on an agent target; sending it on a flow target is a 400.

- `frequencyPenalty` (double, optional)
- `maxTokens` (integer, optional)
- `model` (string, optional)
- `presencePenalty` (double, optional)
- `reasoning` (any, optional)
- `responseFormat` (enum, optional)
  - Allowed values: `default`, `json`, `markdown`, `html`, `xml`
- `seed` (double, optional)
- `temperature` (double, optional)
- `tools` (any, optional)
- `topK` (double, optional)
- `topP` (double, optional)

### EvalOverridesClaudeManaged

Claude Managed variant override. Valid only on a claude_managed agent target; rejected on a standard agent rather than silently ignored.

- `model` (string, optional)
- `systemPrompt` (string, optional)
- `toolPermissions` (map from string to boolean, optional)

### EvalOverridesFlow

Applies to every prompt step of the flow under test. Valid only on a flow target; sending it on an agent target is a 400.

- `frequencyPenalty` (double, optional)
- `maxTokens` (integer, optional)
- `model` (string, optional)
- `presencePenalty` (double, optional)
- `reasoning` (any, optional)
- `responseFormat` (enum, optional)
  - Allowed values: `default`, `json`, `markdown`, `html`, `xml`
- `seed` (double, optional)
- `temperature` (double, optional)
- `tools` (any, optional)
- `topK` (double, optional)
- `topP` (double, optional)

### EvalOverridesSteps

Sampling/model/tool values applied to one override target. Unset fields inherit; a field set here beats the same field on a broader target.

- `frequencyPenalty` (double, optional)
- `maxTokens` (integer, optional)
- `model` (string, optional)
- `presencePenalty` (double, optional)
- `reasoning` (any, optional)
- `responseFormat` (enum, optional)
  - Allowed values: `default`, `json`, `markdown`, `html`, `xml`
- `seed` (double, optional)
- `temperature` (double, optional)
- `tools` (any, optional)
- `topK` (double, optional)
- `topP` (double, optional)

### EvalOverridesSubagentDefaults

Applies to every reachable sub-agent with no `subagents` entry of its own. Does not apply to a dynamically-spawned child, which keeps the model its spawn resolved.

- `frequencyPenalty` (double, optional)
- `maxTokens` (integer, optional)
- `model` (string, optional)
- `presencePenalty` (double, optional)
- `reasoning` (any, optional)
- `responseFormat` (enum, optional)
  - Allowed values: `default`, `json`, `markdown`, `html`, `xml`
- `seed` (double, optional)
- `temperature` (double, optional)
- `tools` (any, optional)
- `topK` (double, optional)
- `topP` (double, optional)

### EvalOverridesSubagents

Sampling/model/tool values applied to one override target. Unset fields inherit; a field set here beats the same field on a broader target.

- `frequencyPenalty` (double, optional)
- `maxTokens` (integer, optional)
- `model` (string, optional)
- `presencePenalty` (double, optional)
- `reasoning` (any, optional)
- `responseFormat` (enum, optional)
  - Allowed values: `default`, `json`, `markdown`, `html`, `xml`
- `seed` (double, optional)
- `temperature` (double, optional)
- `tools` (any, optional)
- `topK` (double, optional)
- `topP` (double, optional)

### ExecutionStreamEventOneOf2Error1

- `code` (string, required)
- `message` (string, required)
- `details` (map from string to any, optional)

### ExecutionStreamEventOneOf4EstimatedContextBreakdownCategories

- `chars` (double, required)
- `estimatedTokens` (double, required)

### ExecutionStreamEventOneOf4FallbackAttemptsItems

- `attempt` (double, required)
- `success` (boolean, required)
- `type` (enum, required)
  - Allowed values: `retry`, `model`, `message`, `flow`
- `error` (string, optional)
- `model` (string, optional)

### ExecutionStreamEventOneOf8FallbackAttemptsItems

- `attempt` (double, required)
- `success` (boolean, required)
- `type` (enum, required)
  - Allowed values: `retry`, `model`, `message`, `flow`
- `error` (string, optional)
- `model` (string, optional)

### ExecutionStreamEventOneOf33ElicitationRequestsItems

- `message` (string, required)
- `mode` (enum, required)
  - Allowed values: `form`, `url`
- `name` (string, required)
- `requestedSchema` (map from string to any, optional)
- `url` (string, optional)

### ExecutionStreamEventOneOf34Error1

- `code` (string, required)
- `message` (string, required)
- `details` (map from string to any, optional)

## Examples

**Request**

```json
{}
```

**SDK Code**

```python
import requests

url = "https://api.runtype.com/v1/eval/stream"

payload = {}
headers = {
    "Authorization": "Bearer <token>",
    "Content-Type": "application/json"
}

response = requests.post(url, json=payload, headers=headers)

print(response.json())
```

```javascript
const url = 'https://api.runtype.com/v1/eval/stream';
const options = {
  method: 'POST',
  headers: {Authorization: 'Bearer <token>', 'Content-Type': 'application/json'},
  body: '{}'
};

try {
  const response = await fetch(url, options);
  const data = await response.json();
  console.log(data);
} catch (error) {
  console.error(error);
}
```

```go
package main

import (
	"fmt"
	"strings"
	"net/http"
	"io"
)

func main() {

	url := "https://api.runtype.com/v1/eval/stream"

	payload := strings.NewReader("{}")

	req, _ := http.NewRequest("POST", url, payload)

	req.Header.Add("Authorization", "Bearer <token>")
	req.Header.Add("Content-Type", "application/json")

	res, _ := http.DefaultClient.Do(req)

	defer res.Body.Close()
	body, _ := io.ReadAll(res.Body)

	fmt.Println(res)
	fmt.Println(string(body))

}
```

```ruby
require 'uri'
require 'net/http'

url = URI("https://api.runtype.com/v1/eval/stream")

http = Net::HTTP.new(url.host, url.port)
http.use_ssl = true

request = Net::HTTP::Post.new(url)
request["Authorization"] = 'Bearer <token>'
request["Content-Type"] = 'application/json'
request.body = "{}"

response = http.request(request)
puts response.read_body
```

```java
import com.mashape.unirest.http.HttpResponse;
import com.mashape.unirest.http.Unirest;

HttpResponse<String> response = Unirest.post("https://api.runtype.com/v1/eval/stream")
  .header("Authorization", "Bearer <token>")
  .header("Content-Type", "application/json")
  .body("{}")
  .asString();
```

```php
<?php
require_once('vendor/autoload.php');

$client = new \GuzzleHttp\Client();

$response = $client->request('POST', 'https://api.runtype.com/v1/eval/stream', [
  'body' => '{}',
  'headers' => [
    'Authorization' => 'Bearer <token>',
    'Content-Type' => 'application/json',
  ],
]);

echo $response->getBody();
```

```csharp
using RestSharp;

var client = new RestClient("https://api.runtype.com/v1/eval/stream");
var request = new RestRequest(Method.POST);
request.AddHeader("Authorization", "Bearer <token>");
request.AddHeader("Content-Type", "application/json");
request.AddParameter("application/json", "{}", ParameterType.RequestBody);
IRestResponse response = client.Execute(request);
```

```swift
import Foundation

let headers = [
  "Authorization": "Bearer <token>",
  "Content-Type": "application/json"
]
let parameters = [] as [String : Any]

let postData = JSONSerialization.data(withJSONObject: parameters, options: [])

let request = NSMutableURLRequest(url: NSURL(string: "https://api.runtype.com/v1/eval/stream")! as URL,
                                        cachePolicy: .useProtocolCachePolicy,
                                    timeoutInterval: 10.0)
request.httpMethod = "POST"
request.allHTTPHeaderFields = headers
request.httpBody = postData as Data

let session = URLSession.shared
let dataTask = session.dataTask(with: request as URLRequest, completionHandler: { (data, response, error) -> Void in
  if (error != nil) {
    print(error as Any)
  } else {
    let httpResponse = response as? HTTPURLResponse
    print(httpResponse)
  }
})

dataTask.resume()
```