> For clean Markdown of any page, append .md to the page URL.
> For a complete documentation index, see https://docs.runtype.com/llms.txt.
> For AI client integration (Claude Code, Cursor, etc.), connect to the MCP server at https://docs.runtype.com/_mcp/server.

# Run eval suite synchronously

POST https://api.runtype.com/v1/eval/run
Content-Type: application/json

Run every case of an eval suite against its target flow/agent and grade the outputs with the suite graders, returning the suite score + per-case gate outcomes in one response. Powers the `runtype eval` CI gate. Saved-suite runs persist by default: the run gets a durable id (`runId`) and per-case scores queryable via GET /eval/runs/\{runId}/scores; pass `virtual: true` to skip persistence. Inline definition runs are always ephemeral. Bounded — suites over the case limit must use the batch path (/eval/submit). Flow and standard-agent targets are supported (in both saved and inline/virtual modes); external and claude\_managed agent targets are not.

Reference: https://docs.runtype.com/api-reference/runtype-api/evals/run-eval-suite-synchronously

## Authentication

- `Authorization` header (bearer token, required) — API key or Clerk session token

## Servers

- `https://api.runtype.com` (Production, default)
- `https://api.runtype-staging.com` (Staging)

## Request

### Body (application/json)

This endpoint expects an object.

- `agent` (V1EvalRunPostRequestBodyContentApplicationJsonSchemaAgent, optional) — Version selector for an Agent target. Supply at most one of alias or versionId; the run resolves it once and every case executes that version.
- `definition` (V1EvalRunPostRequestBodyContentApplicationJsonSchemaDefinition, optional)
- `modelOverride` (string, optional) — Run every case against this model instead of the target’s configured one, without editing the suite, the flow, or the agent. Re-run the same suite once per candidate model and compare the scores to choose a model on graded evidence. Applies to flow and agent targets alike.
- `strict` (boolean, optional) — Strict mode (the `runtype eval --strict` gate): a soft grader miss fails its case like a gate miss. Default false — soft misses are reported per-outcome but do not fail the suite.
- `suiteId` (string, optional)
- `virtual` (boolean, optional) — Skip persistence for a saved-suite run: no run row, no persisted scores. Saved-suite runs persist by default; inline definition runs are always ephemeral regardless of this flag.

## Response

### 200

Suite score + per-case grader outcomes

- `cases` (list of RunEvalResponseCasesItems, required)
- `name` (string, required)
- `passed` (boolean, required) — True when every case passed every grader.
- `passedCases` (integer, required)
- `platformAbortedCases` (integer, required) — Cases the platform aborted before the target could answer. They run no graders and are counted in neither half of score, so totalCases minus this is the denominator score was taken over.
- `runId` (string, required, nullable) — The persisted run id — query its scores via GET /eval/runs/\{runId}/scores. Null for an ephemeral (virtual/inline) run.
- `score` (double, required) — Suite score, 0..1 (passed cases / graded cases). Platform-aborted cases are excluded from both halves.
- `suiteId` (string, required, nullable)
- `targetType` (enum, required)
  - Allowed values: `flow`, `agent`
- `totalCases` (integer, required)
- `agentTargetAlias` (string, optional, nullable) — The release alias the run resolved through, when it resolved through one.
- `agentTargetResolution` (enum, optional) — How the run picked its version. Null on a run that predates release aliases.
  - Allowed values: `alias`, `version`, `legacy-live-row`
- `agentVersionId` (string, optional, nullable) — The exact Agent version every case of this run executed. Null for a legacy run.
- `caseManifestHash` (string, optional, nullable) — SHA-256 over the ordered, normalized case set this run executed.
- `evaluatorFingerprint` (string, optional, nullable) — SHA-256 over the evaluator configuration this run scored with.

## Errors

### 400 Bad Request Error

Validation error (bad definition, unresolved/unsupported target, too many cases)

- `error` (string, required)
- `code` (string, optional) — Stable error code. `insufficient_scope` marks an API key that lacks a scope this request needs.
- `details` (list of ErrorDetailsItems, optional)
- `requiredScopes` (list of string, optional) — On `insufficient_scope`: every API key scope this request needs.

### 401 Unauthorized Error

Unauthorized

- `error` (string, required)
- `code` (string, optional) — Stable error code. `insufficient_scope` marks an API key that lacks a scope this request needs.
- `details` (list of ErrorDetailsItems, optional)
- `requiredScopes` (list of string, optional) — On `insufficient_scope`: every API key scope this request needs.

### 402 Payment Required Error

Daily eval limit exceeded

- `code` (string, required) — Stable error code. DAILY_EVAL_LIMIT: the daily eval count of the plan is used up; retry after `stats.resetsAt`. SPEND_LIMIT_EXCEEDED: the monthly spend cap is reached. CLAUDE_MANAGED_AGENTS_DENIED: the account cannot evaluate a Claude Managed agent.
- `error` (string, required)
- `message` (string, required) — Human-readable reason, including the limit and when it resets.
- `details` (list of EvalQuotaExceededErrorDetailsItems, optional)
- `requiredScopes` (list of string, optional) — On `insufficient_scope`: every API key scope this request needs.
- `stats` (EvalQuotaExceededErrorStats, optional) — Daily eval usage at the moment of a DAILY_EVAL_LIMIT or SPEND_LIMIT_EXCEEDED refusal.

### 403 Forbidden Error

Forbidden

- `error` (string, required)
- `code` (string, optional) — Stable error code. `insufficient_scope` marks an API key that lacks a scope this request needs.
- `details` (list of ErrorDetailsItems, optional)
- `requiredScopes` (list of string, optional) — On `insufficient_scope`: every API key scope this request needs.

### 404 Not Found Error

Suite or target not found

- `error` (string, required)
- `code` (string, optional) — Stable error code. `insufficient_scope` marks an API key that lacks a scope this request needs.
- `details` (list of ErrorDetailsItems, optional)
- `requiredScopes` (list of string, optional) — On `insufficient_scope`: every API key scope this request needs.

### 500 Internal Server Error

Internal server error

- `code` (string, required) — Stable error code: `INTERNAL_ERROR` unless the route documents its own.
- `error` (string, required)
- `details` (list of InternalErrorDetailsItems, optional)
- `requiredScopes` (list of string, optional) — On `insufficient_scope`: every API key scope this request needs.
- `message` (string, optional) — Plain-language guidance: retry, then contact support with the `requestId`.
- `requestId` (string, optional) — The `X-Runtype-Request-ID` of the failed request, for support.

### 503 Service Unavailable Error

Database temporarily unavailable; nothing was read or written. Retry after the `Retry-After` interval (seconds).

- `code` (enum, required) — Stable error code. `insufficient_scope` marks an API key that lacks a scope this request needs.
  - Allowed values: `DATABASE_UNAVAILABLE`
- `error` (string, required)
- `details` (list of DatabaseUnavailableErrorDetailsItems, optional)
- `requiredScopes` (list of string, optional) — On `insufficient_scope`: every API key scope this request needs.
- `requestId` (string, optional) — The `X-Runtype-Request-ID` of the failed request, for support.

## Types

### V1EvalRunPostRequestBodyContentApplicationJsonSchemaAgent

Version selector for an Agent target. Supply at most one of alias or versionId; the run resolves it once and every case executes that version.

- `alias` (string, optional) — Run whichever version this release alias currently points at. Resolved once for the whole run.
- `versionId` (string, optional) — Run this exact immutable version snapshot.

### V1EvalRunPostRequestBodyContentApplicationJsonSchemaDefinition

- `cases` (list of V1EvalRunPostRequestBodyContentApplicationJsonSchemaDefinitionCasesItems, required)
- `name` (string, required)
- `target` (V1EvalRunPostRequestBodyContentApplicationJsonSchemaDefinitionTarget, required)
- `identityDefaults` (V1EvalRunPostRequestBodyContentApplicationJsonSchemaDefinitionIdentityDefaults, optional) — Default identity (endUser / tenant) every case runs as unless the case sets its own. Omitting it on a converge clears the stored default.
- `recordedToolMode` (enum, optional) — How cases carrying toolMocks replay — captured (saved_from_run) or hand-authored alike: "next_step" (default; captured cases only) captures the first tool intent and stops, or "continue" serves each case's toolMocks by tool name and runs past the first action.
  - Allowed values: `next_step`, `continue`
- `recordedToolUnmatchedPolicy` (enum, optional) — For "continue"-mode replays, the unmatched-call policy: "fail" (default) or "stub".
  - Allowed values: `fail`, `stub`
- `virtual` (boolean, optional)

### RunEvalResponseCasesItems

- `errored` (boolean, required)
- `name` (string, required)
- `outcomes` (list of RunEvalResponseCasesItemsOutcomesItems, required)
- `outputExcerpt` (string, required)
- `passed` (boolean, required)
- `error` (string, optional) — Why the case errored (the provider or execution failure text, 2000 chars max). Present only when errored is true and the run reported a cause.
- `failureKind` (enum, optional) — Present when the failure belongs to the platform, not the target: the model gateway went silent and the stall watchdog gave up before the target could answer. Such a case runs no graders and is excluded from the suite score. Absent on an ordinary failure.
  - Allowed values: `platform_stall`

### ErrorDetailsItems

- `message` (string, required)
- `code` (string, optional) — Stable Runtype validation code (e.g. MISSING_REQUIRED_FIELD, INVALID_ENUM).
- `path` (string, optional) — Dotted JSON path to the offending field.
- `suggestion` (string, optional) — Actionable hint (allowed values, accepted range, …).

### EvalQuotaExceededErrorDetailsItems

- `message` (string, required)
- `code` (string, optional) — Stable Runtype validation code (e.g. MISSING_REQUIRED_FIELD, INVALID_ENUM).
- `path` (string, optional) — Dotted JSON path to the offending field.
- `suggestion` (string, optional) — Actionable hint (allowed values, accepted range, …).

### EvalQuotaExceededErrorStats

Daily eval usage at the moment of a DAILY_EVAL_LIMIT or SPEND_LIMIT_EXCEEDED refusal.

- `evalLimit` (integer, required) — Daily eval limit; -1 means unlimited.
- `evalsUsed` (integer, required)
- `isAtLimit` (boolean, required)
- `remaining` (double, required, nullable)
- `resetsAt` (string, required) — ISO 8601 instant the daily count resets (midnight UTC).

### InternalErrorDetailsItems

- `message` (string, required)
- `code` (string, optional) — Stable Runtype validation code (e.g. MISSING_REQUIRED_FIELD, INVALID_ENUM).
- `path` (string, optional) — Dotted JSON path to the offending field.
- `suggestion` (string, optional) — Actionable hint (allowed values, accepted range, …).

### DatabaseUnavailableErrorDetailsItems

- `message` (string, required)
- `code` (string, optional) — Stable Runtype validation code (e.g. MISSING_REQUIRED_FIELD, INVALID_ENUM).
- `path` (string, optional) — Dotted JSON path to the offending field.
- `suggestion` (string, optional) — Actionable hint (allowed values, accepted range, …).

### V1EvalRunPostRequestBodyContentApplicationJsonSchemaDefinitionCasesItems

- `expect` (list of V1EvalRunPostRequestBodyContentApplicationJsonSchemaDefinitionCasesItemsExpectItems, required)
- `name` (string, required)
- `expected` (V1EvalRunPostRequestBodyContentApplicationJsonSchemaDefinitionCasesItemsExpected, optional)
- `input` (V1EvalRunPostRequestBodyContentApplicationJsonSchemaDefinitionCasesItemsInput, optional, default: {})

### V1EvalRunPostRequestBodyContentApplicationJsonSchemaDefinitionTarget

- `agent` (string, optional)
- `flow` (string, optional)

### V1EvalRunPostRequestBodyContentApplicationJsonSchemaDefinitionIdentityDefaults

Default identity (endUser / tenant) every case runs as unless the case sets its own. Omitting it on a converge clears the stored default.

- `endUser` (V1EvalRunPostRequestBodyContentApplicationJsonSchemaDefinitionIdentityDefaultsEndUser, optional) — The end user (the person) the turn runs as. Exposed to tools as _endUser.*, drives memory-profile sharding, and satisfies an end-user-isolated tenancyStrategy. Distinct from _user (the Builder User).
- `tenant` (V1EvalRunPostRequestBodyContentApplicationJsonSchemaDefinitionIdentityDefaultsTenant, optional) — The product tenant (a customer of your product, under your Builder Org) the turn runs within. Exposed to tools as _tenant.*.

### RunEvalResponseCasesItemsOutcomesItems

- `graderIndex` (integer, required)
- `kind` (string, required)
- `passed` (boolean, required)
- `evidence` (list of RunEvalResponseCasesItemsOutcomesItemsEvidenceItems, optional) — AI graders only: transcript entries the verdict rests on — indices into the run's trace steps (message) or tool calls (tool_call).
- `judgeDiagnostics` (RunEvalResponseCasesItemsOutcomesItemsJudgeDiagnostics, optional)
- `reasoning` (string, optional)
- `score` (double, optional)
- `severity` (enum, optional) — Grader severity. A soft miss is reported but only fails the suite under strict mode. Absent ⇒ gate.
  - Allowed values: `gate`, `soft`
- `verdict` (enum, optional) — AI graders only: the judge's binary verdict. insufficient_evidence means the judge declined to guess; it renders as a warning and never fails the case.
  - Allowed values: `pass`, `fail`, `insufficient_evidence`

### V1EvalRunPostRequestBodyContentApplicationJsonSchemaDefinitionCasesItemsExpectItems

- `kind` (string, required)

### V1EvalRunPostRequestBodyContentApplicationJsonSchemaDefinitionCasesItemsExpected

- `facts` (list of string, optional)
- `json` (any, optional)
- `text` (string, optional)

### V1EvalRunPostRequestBodyContentApplicationJsonSchemaDefinitionCasesItemsInput

- `endUser` (V1EvalRunPostRequestBodyContentApplicationJsonSchemaDefinitionCasesItemsInputEndUser, optional) — The end user (the person) the turn runs as. Exposed to tools as _endUser.*, drives memory-profile sharding, and satisfies an end-user-isolated tenancyStrategy. Distinct from _user (the Builder User).
- `messages` (list of V1EvalRunPostRequestBodyContentApplicationJsonSchemaDefinitionCasesItemsInputMessagesItems, optional)
- `tenant` (V1EvalRunPostRequestBodyContentApplicationJsonSchemaDefinitionCasesItemsInputTenant, optional) — The product tenant (a customer of your product, under your Builder Org) the turn runs within. Exposed to tools as _tenant.*.
- `toolMocks` (list of V1EvalRunPostRequestBodyContentApplicationJsonSchemaDefinitionCasesItemsInputToolMocksItems, optional)
- `variables` (map from string to any, optional)

### V1EvalRunPostRequestBodyContentApplicationJsonSchemaDefinitionIdentityDefaultsEndUser

The end user (the person) the turn runs as. Exposed to tools as _endUser.*, drives memory-profile sharding, and satisfies an end-user-isolated tenancyStrategy. Distinct from _user (the Builder User).

- `id` (string, required)
- `email` (string, optional)

### V1EvalRunPostRequestBodyContentApplicationJsonSchemaDefinitionIdentityDefaultsTenant

The product tenant (a customer of your product, under your Builder Org) the turn runs within. Exposed to tools as _tenant.*.

- `id` (string, required)

### RunEvalResponseCasesItemsOutcomesItemsEvidenceItems

- `index` (integer, required)
- `kind` (enum, required)
  - Allowed values: `message`, `tool_call`

### RunEvalResponseCasesItemsOutcomesItemsJudgeDiagnostics

- `attempts` (list of RunEvalResponseCasesItemsOutcomesItemsJudgeDiagnosticsAttemptsItems, required)
- `flowFailure` (enum, optional)
  - Allowed values: `deadline_exhausted`, `execution_failed`, `invalid_response`

### V1EvalRunPostRequestBodyContentApplicationJsonSchemaDefinitionCasesItemsInputEndUser

The end user (the person) the turn runs as. Exposed to tools as _endUser.*, drives memory-profile sharding, and satisfies an end-user-isolated tenancyStrategy. Distinct from _user (the Builder User).

- `id` (string, required)
- `email` (string, optional)

### V1EvalRunPostRequestBodyContentApplicationJsonSchemaDefinitionCasesItemsInputMessagesItems

- `content` (string, required)
- `role` (string, required)

### V1EvalRunPostRequestBodyContentApplicationJsonSchemaDefinitionCasesItemsInputTenant

The product tenant (a customer of your product, under your Builder Org) the turn runs within. Exposed to tools as _tenant.*.

- `id` (string, required)

### V1EvalRunPostRequestBodyContentApplicationJsonSchemaDefinitionCasesItemsInputToolMocksItems

- `output` (V1EvalRunPostRequestBodyContentApplicationJsonSchemaDefinitionCasesItemsInputToolMocksItemsOutput, required, nullable)
- `toolName` (string, required)
- `input` (any, optional)
- `isError` (boolean, optional)
- `sourceToolExecutionId` (string, optional)
- `truncated` (boolean, optional)

### RunEvalResponseCasesItemsOutcomesItemsJudgeDiagnosticsAttemptsItems

- `attempt` (RunEvalResponseCasesItemsOutcomesItemsJudgeDiagnosticsAttemptsItemsAttempt, required)
- `durationMs` (integer, required)
- `requestedModel` (string, required)
- `status` (enum, required)
  - Allowed values: `valid`, `invalid_response`, `model_error`
- `usageObserved` (boolean, required)
- `cacheReadTokens` (integer, optional)
- `cacheWriteTokens` (integer, optional)
- `completionTokens` (integer, optional)
- `finishReason` (enum, optional)
  - Allowed values: `stop`, `length`, `content-filter`, `tool-calls`, `error`, `other`, `unknown`
- `modelError` (enum, optional)
  - Allowed values: `empty_completion`, `model_call_error`
- `parseError` (enum, optional)
  - Allowed values: `empty_completion`, `invalid_json`, `invalid_contract`, `missing_access_assessment`, `invalid_access_assessment`, `output_truncated`
- `promptTokens` (integer, optional)
- `provider` (string, optional)
- `reasoningTokens` (integer, optional)
- `resolvedModel` (string, optional)
- `responseChars` (integer, optional)

### V1EvalRunPostRequestBodyContentApplicationJsonSchemaDefinitionCasesItemsInputToolMocksItemsOutput

- boolean
- double
- string
- list of any
- map from string to any

### RunEvalResponseCasesItemsOutcomesItemsJudgeDiagnosticsAttemptsItemsAttempt

- enum
  - Allowed values: `1`
- enum
  - Allowed values: `2`

## Examples

**Request**

```json
{}
```

**Response**

```json
{
  "cases": [
    {
      "errored": true,
      "name": "string",
      "outcomes": [
        {
          "graderIndex": 1,
          "kind": "string",
          "passed": true,
          "evidence": [
            {
              "index": 1,
              "kind": "message"
            }
          ],
          "judgeDiagnostics": {
            "attempts": [
              {
                "attempt": 1,
                "durationMs": 1,
                "requestedModel": "string",
                "status": "valid",
                "usageObserved": true,
                "cacheReadTokens": 1,
                "cacheWriteTokens": 1,
                "completionTokens": 1,
                "finishReason": "stop",
                "modelError": "empty_completion",
                "parseError": "empty_completion",
                "promptTokens": 1,
                "provider": "string",
                "reasoningTokens": 1,
                "resolvedModel": "string",
                "responseChars": 1
              }
            ],
            "flowFailure": "deadline_exhausted"
          },
          "reasoning": "string",
          "score": 1.1,
          "severity": "gate",
          "verdict": "pass"
        }
      ],
      "outputExcerpt": "string",
      "passed": true,
      "error": "string",
      "failureKind": "platform_stall"
    }
  ],
  "name": "string",
  "passed": true,
  "passedCases": 1,
  "platformAbortedCases": 1,
  "runId": "string",
  "score": 1.1,
  "suiteId": "string",
  "targetType": "flow",
  "totalCases": 1,
  "agentTargetAlias": "string",
  "agentTargetResolution": "alias",
  "agentVersionId": "string",
  "caseManifestHash": "string",
  "evaluatorFingerprint": "string"
}
```

**SDK Code**

```python
import requests

url = "https://api.runtype.com/v1/eval/run"

payload = {}
headers = {
    "Authorization": "Bearer <token>",
    "Content-Type": "application/json"
}

response = requests.post(url, json=payload, headers=headers)

print(response.json())
```

```javascript
const url = 'https://api.runtype.com/v1/eval/run';
const options = {
  method: 'POST',
  headers: {Authorization: 'Bearer <token>', 'Content-Type': 'application/json'},
  body: '{}'
};

try {
  const response = await fetch(url, options);
  const data = await response.json();
  console.log(data);
} catch (error) {
  console.error(error);
}
```

```go
package main

import (
	"fmt"
	"strings"
	"net/http"
	"io"
)

func main() {

	url := "https://api.runtype.com/v1/eval/run"

	payload := strings.NewReader("{}")

	req, _ := http.NewRequest("POST", url, payload)

	req.Header.Add("Authorization", "Bearer <token>")
	req.Header.Add("Content-Type", "application/json")

	res, _ := http.DefaultClient.Do(req)

	defer res.Body.Close()
	body, _ := io.ReadAll(res.Body)

	fmt.Println(res)
	fmt.Println(string(body))

}
```

```ruby
require 'uri'
require 'net/http'

url = URI("https://api.runtype.com/v1/eval/run")

http = Net::HTTP.new(url.host, url.port)
http.use_ssl = true

request = Net::HTTP::Post.new(url)
request["Authorization"] = 'Bearer <token>'
request["Content-Type"] = 'application/json'
request.body = "{}"

response = http.request(request)
puts response.read_body
```

```java
import com.mashape.unirest.http.HttpResponse;
import com.mashape.unirest.http.Unirest;

HttpResponse<String> response = Unirest.post("https://api.runtype.com/v1/eval/run")
  .header("Authorization", "Bearer <token>")
  .header("Content-Type", "application/json")
  .body("{}")
  .asString();
```

```php
<?php
require_once('vendor/autoload.php');

$client = new \GuzzleHttp\Client();

$response = $client->request('POST', 'https://api.runtype.com/v1/eval/run', [
  'body' => '{}',
  'headers' => [
    'Authorization' => 'Bearer <token>',
    'Content-Type' => 'application/json',
  ],
]);

echo $response->getBody();
```

```csharp
using RestSharp;

var client = new RestClient("https://api.runtype.com/v1/eval/run");
var request = new RestRequest(Method.POST);
request.AddHeader("Authorization", "Bearer <token>");
request.AddHeader("Content-Type", "application/json");
request.AddParameter("application/json", "{}", ParameterType.RequestBody);
IRestResponse response = client.Execute(request);
```

```swift
import Foundation

let headers = [
  "Authorization": "Bearer <token>",
  "Content-Type": "application/json"
]
let parameters = [] as [String : Any]

let postData = JSONSerialization.data(withJSONObject: parameters, options: [])

let request = NSMutableURLRequest(url: NSURL(string: "https://api.runtype.com/v1/eval/run")! as URL,
                                        cachePolicy: .useProtocolCachePolicy,
                                    timeoutInterval: 10.0)
request.httpMethod = "POST"
request.allHTTPHeaderFields = headers
request.httpBody = postData as Data

let session = URLSession.shared
let dataTask = session.dataTask(with: request as URLRequest, completionHandler: { (data, response, error) -> Void in
  if (error != nil) {
    print(error as Any)
  } else {
    let httpResponse = response as? HTTPURLResponse
    print(httpResponse)
  }
})

dataTask.resume()
```