> For clean Markdown of any page, append .md to the page URL.
> For a complete documentation index, see https://docs.runtype.com/llms.txt.
> For AI client integration (Claude Code, Cursor, etc.), connect to the MCP server at https://docs.runtype.com/_mcp/server.

# Run eval suite

POST https://api.runtype.com/v1/eval/suites/{id}/run
Content-Type: application/json

Run every enabled case of the suite against its target and grade the outputs with the suite graders. Suites within the synchronous case limit (50) run inline and return the full scored result; larger suites are queued as a durable run (202) that is graded when it completes — poll GET /eval/runs/\{runId}/scores. Both paths persist per-case scores to the same run history.

Reference: https://docs.runtype.com/api-reference/runtype-api/evals/run-eval-suite

## Authentication

- `Authorization` header (bearer token, required) — API key or Clerk session token

## Servers

- `https://api.runtype.com` (Production, default)
- `https://api.runtype-staging.com` (Staging)

## Request

### Path parameters

- `id` (string, required)

### Body (application/json)

This endpoint expects an object.

- `agent` (EvalAgentSelector, optional) — Version selector for an Agent-target suite. Supply at most one of alias or versionId; omit it to run the saved configuration.
- `mode` (enum, optional) — Force the execution path. Default: sync for suites within the synchronous case limit (50), batch above it.
  - Allowed values: `sync`, `batch`
- `modelOverride` (string, optional) — Run every case against this model instead of the target’s configured one, without editing the suite or its target. Re-run the same suite once per candidate model and compare scores to pick a model on graded evidence. Applies to both flow and agent targets.
- `strict` (boolean, optional) — Strict grading: a soft grader miss fails its case like a gate miss.

## Response

### 200

Suite ran synchronously — score + per-case grader outcomes

- `cases` (list of RunEvalResponseCasesItems, required)
- `name` (string, required)
- `passed` (boolean, required) — True when every case passed every grader.
- `passedCases` (integer, required)
- `platformAbortedCases` (integer, required) — Cases the platform aborted before the target could answer. They run no graders and are counted in neither half of score, so totalCases minus this is the denominator score was taken over.
- `runId` (string, required, nullable) — The persisted run id — query its scores via GET /eval/runs/\{runId}/scores. Null for an ephemeral (virtual/inline) run.
- `score` (double, required) — Suite score, 0..1 (passed cases / graded cases). Platform-aborted cases are excluded from both halves.
- `suiteId` (string, required, nullable)
- `targetType` (enum, required)
  - Allowed values: `flow`, `agent`
- `totalCases` (integer, required)
- `agentTargetAlias` (string, optional, nullable) — The release alias the run resolved through, when it resolved through one.
- `agentTargetResolution` (enum, optional) — How the run picked its version. Null on a run that predates release aliases.
  - Allowed values: `alias`, `version`, `legacy-live-row`
- `agentVersionId` (string, optional, nullable) — The exact Agent version every case of this run executed. Null for a legacy run.
- `caseManifestHash` (string, optional, nullable) — SHA-256 over the ordered, normalized case set this run executed.
- `evaluatorFingerprint` (string, optional, nullable) — SHA-256 over the evaluator configuration this run scored with.

### 202

Suite queued as a durable run (scored on completion)

- `mode` (enum, required)
  - Allowed values: `batch`
- `name` (string, required)
- `runId` (string, required) — The queued run id. Poll GET /eval/\{runId}/results for progress and GET /eval/runs/\{runId}/scores for grader scores once completed.
- `status` (string, required)
- `suiteId` (string, required)
- `totalCases` (integer, required)
- `agentTargetAlias` (string, optional, nullable) — The release alias the run resolved through, when it resolved through one.
- `agentTargetResolution` (enum, optional) — How the run picked its version. Null on a run that predates release aliases.
  - Allowed values: `alias`, `version`, `legacy-live-row`
- `agentVersionId` (string, optional, nullable) — The exact Agent version every case of this run executed. Null for a legacy run.
- `caseManifestHash` (string, optional, nullable) — SHA-256 over the ordered, normalized case set this run executed.
- `evaluatorFingerprint` (string, optional, nullable) — SHA-256 over the evaluator configuration this run scored with.

## Errors

### 400 Bad Request Error

Invalid request

- `error` (string, required)
- `code` (string, optional) — Stable error code. `insufficient_scope` marks an API key that lacks a scope this request needs.
- `details` (list of ErrorDetailsItems, optional)
- `requiredScopes` (list of string, optional) — On `insufficient_scope`: every API key scope this request needs.

### 401 Unauthorized Error

Unauthorized

- `error` (string, required)
- `code` (string, optional) — Stable error code. `insufficient_scope` marks an API key that lacks a scope this request needs.
- `details` (list of ErrorDetailsItems, optional)
- `requiredScopes` (list of string, optional) — On `insufficient_scope`: every API key scope this request needs.

### 402 Payment Required Error

Daily eval limit exceeded

- `code` (string, required) — Stable error code. DAILY_EVAL_LIMIT: the daily eval count of the plan is used up; retry after `stats.resetsAt`. SPEND_LIMIT_EXCEEDED: the monthly spend cap is reached. CLAUDE_MANAGED_AGENTS_DENIED: the account cannot evaluate a Claude Managed agent.
- `error` (string, required)
- `message` (string, required) — Human-readable reason, including the limit and when it resets.
- `details` (list of EvalQuotaExceededErrorDetailsItems, optional)
- `requiredScopes` (list of string, optional) — On `insufficient_scope`: every API key scope this request needs.
- `stats` (EvalQuotaExceededErrorStats, optional) — Daily eval usage at the moment of a DAILY_EVAL_LIMIT or SPEND_LIMIT_EXCEEDED refusal.

### 403 Forbidden Error

Forbidden

- `error` (string, required)
- `code` (string, optional) — Stable error code. `insufficient_scope` marks an API key that lacks a scope this request needs.
- `details` (list of ErrorDetailsItems, optional)
- `requiredScopes` (list of string, optional) — On `insufficient_scope`: every API key scope this request needs.

### 404 Not Found Error

Not found, a version selector that names no pointer (alias_not_found, alias_archived, alias_expired, version_not_found), or a saved agent with no runnable live version (AGENT_NOT_DEPLOYED)

- `error` (string, required)
- `code` (string, optional) — Stable error code. `insufficient_scope` marks an API key that lacks a scope this request needs.
- `details` (list of ErrorDetailsItems, optional)
- `requiredScopes` (list of string, optional) — On `insufficient_scope`: every API key scope this request needs.

### 422 Unprocessable Entity Error

A durable run of a flow target the batch host cannot run (RUNTIME_LANE_INELIGIBLE, with `reasons`). Nothing is queued and no eval run is counted.

- `code` (enum, required) — The flow uses capabilities the batch host cannot run.
  - Allowed values: `RUNTIME_LANE_INELIGIBLE`
- `error` (string, required)
- `reasons` (list of string, required) — Stable reason codes: `ineligible:*` capabilities (e.g. `ineligible:wait-until`), or `eval-override:*` for an eval override the batch host cannot apply.
- `details` (list of BatchLaneRefusalErrorDetailsItems, optional)
- `requiredScopes` (list of string, optional) — On `insufficient_scope`: every API key scope this request needs.

### 500 Internal Server Error

Internal server error

- `code` (string, required) — Stable error code: `INTERNAL_ERROR` unless the route documents its own.
- `error` (string, required)
- `details` (list of InternalErrorDetailsItems, optional)
- `requiredScopes` (list of string, optional) — On `insufficient_scope`: every API key scope this request needs.
- `message` (string, optional) — Plain-language guidance: retry, then contact support with the `requestId`.
- `requestId` (string, optional) — The `X-Runtype-Request-ID` of the failed request, for support.

### 503 Service Unavailable Error

Database temporarily unavailable; nothing was read or written. Retry after the `Retry-After` interval (seconds).

- `code` (enum, required) — Stable error code. `insufficient_scope` marks an API key that lacks a scope this request needs.
  - Allowed values: `DATABASE_UNAVAILABLE`
- `error` (string, required)
- `details` (list of DatabaseUnavailableErrorDetailsItems, optional)
- `requiredScopes` (list of string, optional) — On `insufficient_scope`: every API key scope this request needs.
- `requestId` (string, optional) — The `X-Runtype-Request-ID` of the failed request, for support.

## Types

### EvalAgentSelector

Version selector for an Agent-target suite. Supply at most one of alias or versionId; omit it to run the saved configuration.

- `alias` (string, optional) — Run whichever version this release alias currently points at. Resolved once for the whole run.
- `versionId` (string, optional) — Run this exact immutable version snapshot.

### RunEvalResponseCasesItems

- `errored` (boolean, required)
- `name` (string, required)
- `outcomes` (list of RunEvalResponseCasesItemsOutcomesItems, required)
- `outputExcerpt` (string, required)
- `passed` (boolean, required)
- `error` (string, optional) — Why the case errored (the provider or execution failure text, 2000 chars max). Present only when errored is true and the run reported a cause.
- `failureKind` (enum, optional) — Present when the failure belongs to the platform, not the target: the model gateway went silent and the stall watchdog gave up before the target could answer. Such a case runs no graders and is excluded from the suite score. Absent on an ordinary failure.
  - Allowed values: `platform_stall`

### ErrorDetailsItems

- `message` (string, required)
- `code` (string, optional) — Stable Runtype validation code (e.g. MISSING_REQUIRED_FIELD, INVALID_ENUM).
- `path` (string, optional) — Dotted JSON path to the offending field.
- `suggestion` (string, optional) — Actionable hint (allowed values, accepted range, …).

### EvalQuotaExceededErrorDetailsItems

- `message` (string, required)
- `code` (string, optional) — Stable Runtype validation code (e.g. MISSING_REQUIRED_FIELD, INVALID_ENUM).
- `path` (string, optional) — Dotted JSON path to the offending field.
- `suggestion` (string, optional) — Actionable hint (allowed values, accepted range, …).

### EvalQuotaExceededErrorStats

Daily eval usage at the moment of a DAILY_EVAL_LIMIT or SPEND_LIMIT_EXCEEDED refusal.

- `evalLimit` (integer, required) — Daily eval limit; -1 means unlimited.
- `evalsUsed` (integer, required)
- `isAtLimit` (boolean, required)
- `remaining` (double, required, nullable)
- `resetsAt` (string, required) — ISO 8601 instant the daily count resets (midnight UTC).

### BatchLaneRefusalErrorDetailsItems

- `message` (string, required)
- `code` (string, optional) — Stable Runtype validation code (e.g. MISSING_REQUIRED_FIELD, INVALID_ENUM).
- `path` (string, optional) — Dotted JSON path to the offending field.
- `suggestion` (string, optional) — Actionable hint (allowed values, accepted range, …).

### InternalErrorDetailsItems

- `message` (string, required)
- `code` (string, optional) — Stable Runtype validation code (e.g. MISSING_REQUIRED_FIELD, INVALID_ENUM).
- `path` (string, optional) — Dotted JSON path to the offending field.
- `suggestion` (string, optional) — Actionable hint (allowed values, accepted range, …).

### DatabaseUnavailableErrorDetailsItems

- `message` (string, required)
- `code` (string, optional) — Stable Runtype validation code (e.g. MISSING_REQUIRED_FIELD, INVALID_ENUM).
- `path` (string, optional) — Dotted JSON path to the offending field.
- `suggestion` (string, optional) — Actionable hint (allowed values, accepted range, …).

### RunEvalResponseCasesItemsOutcomesItems

- `graderIndex` (integer, required)
- `kind` (string, required)
- `passed` (boolean, required)
- `evidence` (list of RunEvalResponseCasesItemsOutcomesItemsEvidenceItems, optional) — AI graders only: transcript entries the verdict rests on — indices into the run's trace steps (message) or tool calls (tool_call).
- `judgeDiagnostics` (RunEvalResponseCasesItemsOutcomesItemsJudgeDiagnostics, optional)
- `reasoning` (string, optional)
- `score` (double, optional)
- `severity` (enum, optional) — Grader severity. A soft miss is reported but only fails the suite under strict mode. Absent ⇒ gate.
  - Allowed values: `gate`, `soft`
- `verdict` (enum, optional) — AI graders only: the judge's binary verdict. insufficient_evidence means the judge declined to guess; it renders as a warning and never fails the case.
  - Allowed values: `pass`, `fail`, `insufficient_evidence`

### RunEvalResponseCasesItemsOutcomesItemsEvidenceItems

- `index` (integer, required)
- `kind` (enum, required)
  - Allowed values: `message`, `tool_call`

### RunEvalResponseCasesItemsOutcomesItemsJudgeDiagnostics

- `attempts` (list of RunEvalResponseCasesItemsOutcomesItemsJudgeDiagnosticsAttemptsItems, required)
- `flowFailure` (enum, optional)
  - Allowed values: `deadline_exhausted`, `execution_failed`, `invalid_response`

### RunEvalResponseCasesItemsOutcomesItemsJudgeDiagnosticsAttemptsItems

- `attempt` (RunEvalResponseCasesItemsOutcomesItemsJudgeDiagnosticsAttemptsItemsAttempt, required)
- `durationMs` (integer, required)
- `requestedModel` (string, required)
- `status` (enum, required)
  - Allowed values: `valid`, `invalid_response`, `model_error`
- `usageObserved` (boolean, required)
- `cacheReadTokens` (integer, optional)
- `cacheWriteTokens` (integer, optional)
- `completionTokens` (integer, optional)
- `finishReason` (enum, optional)
  - Allowed values: `stop`, `length`, `content-filter`, `tool-calls`, `error`, `other`, `unknown`
- `modelError` (enum, optional)
  - Allowed values: `empty_completion`, `model_call_error`
- `parseError` (enum, optional)
  - Allowed values: `empty_completion`, `invalid_json`, `invalid_contract`, `missing_access_assessment`, `invalid_access_assessment`, `output_truncated`
- `promptTokens` (integer, optional)
- `provider` (string, optional)
- `reasoningTokens` (integer, optional)
- `resolvedModel` (string, optional)
- `responseChars` (integer, optional)

### RunEvalResponseCasesItemsOutcomesItemsJudgeDiagnosticsAttemptsItemsAttempt

- enum
  - Allowed values: `1`
- enum
  - Allowed values: `2`

## Examples

### Example 1

**Request**

```json
{}
```

**Response**

```json
{
  "cases": [
    {
      "errored": true,
      "name": "string",
      "outcomes": [
        {
          "graderIndex": 1,
          "kind": "string",
          "passed": true,
          "evidence": [
            {
              "index": 1,
              "kind": "message"
            }
          ],
          "judgeDiagnostics": {
            "attempts": [
              {
                "attempt": 1,
                "durationMs": 1,
                "requestedModel": "string",
                "status": "valid",
                "usageObserved": true,
                "cacheReadTokens": 1,
                "cacheWriteTokens": 1,
                "completionTokens": 1,
                "finishReason": "stop",
                "modelError": "empty_completion",
                "parseError": "empty_completion",
                "promptTokens": 1,
                "provider": "string",
                "reasoningTokens": 1,
                "resolvedModel": "string",
                "responseChars": 1
              }
            ],
            "flowFailure": "deadline_exhausted"
          },
          "reasoning": "string",
          "score": 1.1,
          "severity": "gate",
          "verdict": "pass"
        }
      ],
      "outputExcerpt": "string",
      "passed": true,
      "error": "string",
      "failureKind": "platform_stall"
    }
  ],
  "name": "string",
  "passed": true,
  "passedCases": 1,
  "platformAbortedCases": 1,
  "runId": "string",
  "score": 1.1,
  "suiteId": "string",
  "targetType": "flow",
  "totalCases": 1,
  "agentTargetAlias": "string",
  "agentTargetResolution": "alias",
  "agentVersionId": "string",
  "caseManifestHash": "string",
  "evaluatorFingerprint": "string"
}
```

**SDK Code**

```python
import requests

url = "https://api.runtype.com/v1/eval/suites/id/run"

payload = {}
headers = {
    "Authorization": "Bearer <token>",
    "Content-Type": "application/json"
}

response = requests.post(url, json=payload, headers=headers)

print(response.json())
```

```javascript
const url = 'https://api.runtype.com/v1/eval/suites/id/run';
const options = {
  method: 'POST',
  headers: {Authorization: 'Bearer <token>', 'Content-Type': 'application/json'},
  body: '{}'
};

try {
  const response = await fetch(url, options);
  const data = await response.json();
  console.log(data);
} catch (error) {
  console.error(error);
}
```

```go
package main

import (
	"fmt"
	"strings"
	"net/http"
	"io"
)

func main() {

	url := "https://api.runtype.com/v1/eval/suites/id/run"

	payload := strings.NewReader("{}")

	req, _ := http.NewRequest("POST", url, payload)

	req.Header.Add("Authorization", "Bearer <token>")
	req.Header.Add("Content-Type", "application/json")

	res, _ := http.DefaultClient.Do(req)

	defer res.Body.Close()
	body, _ := io.ReadAll(res.Body)

	fmt.Println(res)
	fmt.Println(string(body))

}
```

```ruby
require 'uri'
require 'net/http'

url = URI("https://api.runtype.com/v1/eval/suites/id/run")

http = Net::HTTP.new(url.host, url.port)
http.use_ssl = true

request = Net::HTTP::Post.new(url)
request["Authorization"] = 'Bearer <token>'
request["Content-Type"] = 'application/json'
request.body = "{}"

response = http.request(request)
puts response.read_body
```

```java
import com.mashape.unirest.http.HttpResponse;
import com.mashape.unirest.http.Unirest;

HttpResponse<String> response = Unirest.post("https://api.runtype.com/v1/eval/suites/id/run")
  .header("Authorization", "Bearer <token>")
  .header("Content-Type", "application/json")
  .body("{}")
  .asString();
```

```php
<?php
require_once('vendor/autoload.php');

$client = new \GuzzleHttp\Client();

$response = $client->request('POST', 'https://api.runtype.com/v1/eval/suites/id/run', [
  'body' => '{}',
  'headers' => [
    'Authorization' => 'Bearer <token>',
    'Content-Type' => 'application/json',
  ],
]);

echo $response->getBody();
```

```csharp
using RestSharp;

var client = new RestClient("https://api.runtype.com/v1/eval/suites/id/run");
var request = new RestRequest(Method.POST);
request.AddHeader("Authorization", "Bearer <token>");
request.AddHeader("Content-Type", "application/json");
request.AddParameter("application/json", "{}", ParameterType.RequestBody);
IRestResponse response = client.Execute(request);
```

```swift
import Foundation

let headers = [
  "Authorization": "Bearer <token>",
  "Content-Type": "application/json"
]
let parameters = [] as [String : Any]

let postData = JSONSerialization.data(withJSONObject: parameters, options: [])

let request = NSMutableURLRequest(url: NSURL(string: "https://api.runtype.com/v1/eval/suites/id/run")! as URL,
                                        cachePolicy: .useProtocolCachePolicy,
                                    timeoutInterval: 10.0)
request.httpMethod = "POST"
request.allHTTPHeaderFields = headers
request.httpBody = postData as Data

let session = URLSession.shared
let dataTask = session.dataTask(with: request as URLRequest, completionHandler: { (data, response, error) -> Void in
  if (error != nil) {
    print(error as Any)
  } else {
    let httpResponse = response as? HTTPURLResponse
    print(httpResponse)
  }
})

dataTask.resume()
```

### Example 2

**Request**

```json
{}
```

**Response**

```json
{
  "cases": [
    {
      "errored": true,
      "name": "string",
      "outcomes": [
        {
          "graderIndex": 1,
          "kind": "string",
          "passed": true,
          "evidence": [
            {
              "index": 1,
              "kind": "message"
            }
          ],
          "judgeDiagnostics": {
            "attempts": [
              {
                "attempt": 1,
                "durationMs": 1,
                "requestedModel": "string",
                "status": "valid",
                "usageObserved": true,
                "cacheReadTokens": 1,
                "cacheWriteTokens": 1,
                "completionTokens": 1,
                "finishReason": "stop",
                "modelError": "empty_completion",
                "parseError": "empty_completion",
                "promptTokens": 1,
                "provider": "string",
                "reasoningTokens": 1,
                "resolvedModel": "string",
                "responseChars": 1
              }
            ],
            "flowFailure": "deadline_exhausted"
          },
          "reasoning": "string",
          "score": 1.1,
          "severity": "gate",
          "verdict": "pass"
        }
      ],
      "outputExcerpt": "string",
      "passed": true,
      "error": "string",
      "failureKind": "platform_stall"
    }
  ],
  "name": "string",
  "passed": true,
  "passedCases": 1,
  "platformAbortedCases": 1,
  "runId": "string",
  "score": 1.1,
  "suiteId": "string",
  "targetType": "flow",
  "totalCases": 1,
  "agentTargetAlias": "string",
  "agentTargetResolution": "alias",
  "agentVersionId": "string",
  "caseManifestHash": "string",
  "evaluatorFingerprint": "string"
}
```

**SDK Code**

```python
import requests

url = "https://api.runtype.com/v1/eval/suites/id/run"

payload = {}
headers = {
    "Authorization": "Bearer <token>",
    "Content-Type": "application/json"
}

response = requests.post(url, json=payload, headers=headers)

print(response.json())
```

```javascript
const url = 'https://api.runtype.com/v1/eval/suites/id/run';
const options = {
  method: 'POST',
  headers: {Authorization: 'Bearer <token>', 'Content-Type': 'application/json'},
  body: '{}'
};

try {
  const response = await fetch(url, options);
  const data = await response.json();
  console.log(data);
} catch (error) {
  console.error(error);
}
```

```go
package main

import (
	"fmt"
	"strings"
	"net/http"
	"io"
)

func main() {

	url := "https://api.runtype.com/v1/eval/suites/id/run"

	payload := strings.NewReader("{}")

	req, _ := http.NewRequest("POST", url, payload)

	req.Header.Add("Authorization", "Bearer <token>")
	req.Header.Add("Content-Type", "application/json")

	res, _ := http.DefaultClient.Do(req)

	defer res.Body.Close()
	body, _ := io.ReadAll(res.Body)

	fmt.Println(res)
	fmt.Println(string(body))

}
```

```ruby
require 'uri'
require 'net/http'

url = URI("https://api.runtype.com/v1/eval/suites/id/run")

http = Net::HTTP.new(url.host, url.port)
http.use_ssl = true

request = Net::HTTP::Post.new(url)
request["Authorization"] = 'Bearer <token>'
request["Content-Type"] = 'application/json'
request.body = "{}"

response = http.request(request)
puts response.read_body
```

```java
import com.mashape.unirest.http.HttpResponse;
import com.mashape.unirest.http.Unirest;

HttpResponse<String> response = Unirest.post("https://api.runtype.com/v1/eval/suites/id/run")
  .header("Authorization", "Bearer <token>")
  .header("Content-Type", "application/json")
  .body("{}")
  .asString();
```

```php
<?php
require_once('vendor/autoload.php');

$client = new \GuzzleHttp\Client();

$response = $client->request('POST', 'https://api.runtype.com/v1/eval/suites/id/run', [
  'body' => '{}',
  'headers' => [
    'Authorization' => 'Bearer <token>',
    'Content-Type' => 'application/json',
  ],
]);

echo $response->getBody();
```

```csharp
using RestSharp;

var client = new RestClient("https://api.runtype.com/v1/eval/suites/id/run");
var request = new RestRequest(Method.POST);
request.AddHeader("Authorization", "Bearer <token>");
request.AddHeader("Content-Type", "application/json");
request.AddParameter("application/json", "{}", ParameterType.RequestBody);
IRestResponse response = client.Execute(request);
```

```swift
import Foundation

let headers = [
  "Authorization": "Bearer <token>",
  "Content-Type": "application/json"
]
let parameters = [] as [String : Any]

let postData = JSONSerialization.data(withJSONObject: parameters, options: [])

let request = NSMutableURLRequest(url: NSURL(string: "https://api.runtype.com/v1/eval/suites/id/run")! as URL,
                                        cachePolicy: .useProtocolCachePolicy,
                                    timeoutInterval: 10.0)
request.httpMethod = "POST"
request.allHTTPHeaderFields = headers
request.httpBody = postData as Data

let session = URLSession.shared
let dataTask = session.dataTask(with: request as URLRequest, completionHandler: { (data, response, error) -> Void in
  if (error != nil) {
    print(error as Any)
  } else {
    let httpResponse = response as? HTTPURLResponse
    print(httpResponse)
  }
})

dataTask.resume()
```