> For clean Markdown of any page, append .md to the page URL.
> For a complete documentation index, see https://docs.runtype.com/llms.txt.
> For AI client integration (Claude Code, Cursor, etc.), connect to the MCP server at https://docs.runtype.com/_mcp/server.

# Create eval suite

POST https://api.runtype.com/v1/eval/suites
Content-Type: application/json

Create an eval suite (named test cases + graders) attached to one flow or agent, optionally with initial cases. Suites created here are dashboard/API-managed; config-as-code suites converge via POST /eval/ensure instead.

Reference: https://docs.runtype.com/api-reference/runtype-api/evals/create-eval-suite

## Authentication

- `Authorization` header (bearer token, required) — API key or Clerk session token

## Servers

- `https://api.runtype.com` (Production, default)
- `https://api.runtype-staging.com` (Staging)

## Request

### Body (application/json)

This endpoint expects an object.

- `name` (string, required)
- `agentId` (string, optional)
- `cases` (list of V1EvalSuitesPostRequestBodyContentApplicationJsonSchemaCasesItems, optional)
- `description` (string, optional)
- `flowId` (string, optional)
- `graders` (list of V1EvalSuitesPostRequestBodyContentApplicationJsonSchemaGradersItems, optional, default: []) — Ordered grader configs applied to every case (suite-level graders).
- `identityDefaults` (V1EvalSuitesPostRequestBodyContentApplicationJsonSchemaIdentityDefaults, optional) — Default identity (endUser / tenant) every case runs as unless the case sets its own. Required (or per-case identity) when the agent sets config.tenancyStrategy. No credentials: the identity is asserted by the suite author.
- `isDefault` (boolean, optional) — Make this the default eval for its target (clears the flag on any other suite for the same target).
- `recordedToolMode` (enum, optional) — How captured (saved_from_run) cases replay. "next_step" (default) captures the first tool intent and stops; "continue" serves recorded toolMocks and runs past the first action.
  - Allowed values: `next_step`, `continue`
- `recordedToolUnmatchedPolicy` (enum, optional) — For "continue"-mode replays, what to do when a tool call has no matching mock: "fail" (default, divergence is the signal) or "stub" (canned success so the run keeps going). A replay never reaches the real tool.
  - Allowed values: `fail`, `stub`

## Response

### 201

Suite created

- `agentId` (string, required, nullable)
- `baselineBatchExecutionId` (string, required, nullable)
- `caseCount` (integer, required)
- `cases` (list of EvalCase, required)
- `createdAt` (string, required)
- `description` (string, required, nullable)
- `enabledCaseCount` (integer, required)
- `flowId` (string, required, nullable)
- `graders` (list of any, required) — Ordered GraderConfig[] applied to every case.
- `id` (string, required)
- `identityDefaults` (EvalSuiteDetailIdentityDefaults, required, nullable) — Default identity (endUser / tenant) cases run as unless they set their own; null when unset.
- `isDefault` (boolean, required)
- `judgeAgreement` (EvalSuiteDetailJudgeAgreement, required, nullable) — The judge-trust corpus (E6 calibration): how often the AI grader agrees with your reviews, surfaced as a plain fraction. Null until at least one verdict has been reviewed (POST /eval/scores/\{scoreId}/review).
- `lastModifiedSource` (string, required, nullable) — Provenance of the last Definition write. 'sdk' / 'terraform' mean the suite is managed in code (converged by ensure on deploy).
- `latestRun` (EvalSuiteLatestRun, required)
- `name` (string, required)
- `recordedToolMode` (enum, required) — How captured cases replay: "next_step" (default when null) or "continue".
  - Allowed values: `next_step`, `continue`
- `recordedToolUnmatchedPolicy` (enum, required) — Unmatched-call policy for "continue"-mode replays: "fail" (default when null) or "stub".
  - Allowed values: `fail`, `stub`
- `targetName` (string, required, nullable) — Display name of the target flow/agent; null when the target row is gone.
- `targetType` (enum, required)
  - Allowed values: `flow`, `agent`
- `updatedAt` (string, required)
- `warnings` (list of EvalSuiteDetailWarningsItems, optional) — Non-blocking findings about this suite. `AGENT_NOT_DEPLOYED` means the target agent has no runnable `live` version, so a run that names no version selector refuses once the live default is on for the organization.

## Errors

### 400 Bad Request Error

Invalid request

- `error` (string, required)
- `code` (string, optional) — Stable error code. `insufficient_scope` marks an API key that lacks a scope this request needs.
- `details` (list of ErrorDetailsItems, optional)
- `requiredScopes` (list of string, optional) — On `insufficient_scope`: every API key scope this request needs.

### 401 Unauthorized Error

Unauthorized

- `error` (string, required)
- `code` (string, optional) — Stable error code. `insufficient_scope` marks an API key that lacks a scope this request needs.
- `details` (list of ErrorDetailsItems, optional)
- `requiredScopes` (list of string, optional) — On `insufficient_scope`: every API key scope this request needs.

### 403 Forbidden Error

Forbidden

- `error` (string, required)
- `code` (string, optional) — Stable error code. `insufficient_scope` marks an API key that lacks a scope this request needs.
- `details` (list of ErrorDetailsItems, optional)
- `requiredScopes` (list of string, optional) — On `insufficient_scope`: every API key scope this request needs.

### 404 Not Found Error

Not found

- `error` (string, required)
- `code` (string, optional) — Stable error code. `insufficient_scope` marks an API key that lacks a scope this request needs.
- `details` (list of ErrorDetailsItems, optional)
- `requiredScopes` (list of string, optional) — On `insufficient_scope`: every API key scope this request needs.

### 500 Internal Server Error

Internal server error

- `error` (string, required)
- `code` (string, optional) — Stable error code. `insufficient_scope` marks an API key that lacks a scope this request needs.
- `details` (list of ErrorDetailsItems, optional)
- `requiredScopes` (list of string, optional) — On `insufficient_scope`: every API key scope this request needs.

## Types

### V1EvalSuitesPostRequestBodyContentApplicationJsonSchemaCasesItems

- `name` (string, required)
- `enabled` (boolean, optional)
- `expected` (V1EvalSuitesPostRequestBodyContentApplicationJsonSchemaCasesItemsExpected, optional)
- `graders` (list of V1EvalSuitesPostRequestBodyContentApplicationJsonSchemaCasesItemsGradersItems, optional) — Per-case graders. When set, they REPLACE the suite-level graders for this case (not appended). Omit to score the case with the suite graders.
- `input` (V1EvalSuitesPostRequestBodyContentApplicationJsonSchemaCasesItemsInput, optional, default: {})
- `notes` (string, optional)
- `origin` (enum, optional) — Case provenance. Defaults to "manual" when omitted. "ensure" is reserved for config-as-code converges.
  - Allowed values: `manual`, `saved_from_run`, `generated`, `imported`, `fpo`

### V1EvalSuitesPostRequestBodyContentApplicationJsonSchemaGradersItems

- `kind` (string, required)

### V1EvalSuitesPostRequestBodyContentApplicationJsonSchemaIdentityDefaults

Default identity (endUser / tenant) every case runs as unless the case sets its own. Required (or per-case identity) when the agent sets config.tenancyStrategy. No credentials: the identity is asserted by the suite author.

- `endUser` (V1EvalSuitesPostRequestBodyContentApplicationJsonSchemaIdentityDefaultsEndUser, optional) — The end user (the person) the turn runs as. Exposed to tools as _endUser.*, drives memory-profile sharding, and satisfies an end-user-isolated tenancyStrategy. Distinct from _user (the Builder User).
- `tenant` (V1EvalSuitesPostRequestBodyContentApplicationJsonSchemaIdentityDefaultsTenant, optional) — The product tenant (a customer of your product, under your Builder Org) the turn runs within. Exposed to tools as _tenant.*.

### EvalCase

- `createdAt` (string, required)
- `enabled` (boolean, required)
- `graders` (list of any, required, nullable) — Per-case graders that REPLACE the suite graders for this case, or null when the case uses the suite graders.
- `id` (string, required)
- `name` (string, required)
- `notes` (string, required, nullable)
- `origin` (string, required)
- `suiteId` (string, required)
- `updatedAt` (string, required)
- `expected` (any, optional)
- `input` (any, optional) — \{ variables?, messages?, toolMocks?, endUser?, tenant?, record? } — what the case feeds the target, the identity it runs as, and the record it runs on.

### EvalSuiteDetailIdentityDefaults

Default identity (endUser / tenant) cases run as unless they set their own; null when unset.

- `endUser` (EvalSuiteDetailIdentityDefaultsEndUser, optional) — The end user (the person) the turn runs as. Exposed to tools as _endUser.*, drives memory-profile sharding, and satisfies an end-user-isolated tenancyStrategy. Distinct from _user (the Builder User).
- `tenant` (EvalSuiteDetailIdentityDefaultsTenant, optional) — The product tenant (a customer of your product, under your Builder Org) the turn runs within. Exposed to tools as _tenant.*.

### EvalSuiteDetailJudgeAgreement

The judge-trust corpus (E6 calibration): how often the AI grader agrees with your reviews, surfaced as a plain fraction. Null until at least one verdict has been reviewed (POST /eval/scores/\{scoreId}/review).

- `agree` (integer, required) — Reviewed AI-grader verdicts the human agreed with.
- `total` (integer, required) — All reviewed AI-grader verdicts across the suite's runs.

### EvalSuiteLatestRun

- `completedAt` (string, required, nullable)
- `passedCases` (integer, required, nullable)
- `platformAbortedCases` (integer, required, nullable) — Cases the platform aborted before the target could answer. They run no graders and are counted in neither half of score, so totalCases minus this is the denominator score was taken over.
- `runId` (string, required)
- `score` (double, required, nullable) — Suite score of the run, 0..1; null while unscored (in flight / unscoreable).
- `startedAt` (string, required)
- `status` (string, required)
- `totalCases` (integer, required, nullable)

### EvalSuiteDetailWarningsItems

- `code` (string, required)
- `message` (string, required)
- `agentId` (string, optional)

### ErrorDetailsItems

- `message` (string, required)
- `code` (string, optional) — Stable Runtype validation code (e.g. MISSING_REQUIRED_FIELD, INVALID_ENUM).
- `path` (string, optional) — Dotted JSON path to the offending field.
- `suggestion` (string, optional) — Actionable hint (allowed values, accepted range, …).

### V1EvalSuitesPostRequestBodyContentApplicationJsonSchemaCasesItemsExpected

- `facts` (list of string, optional)
- `json` (any, optional)
- `text` (string, optional)

### V1EvalSuitesPostRequestBodyContentApplicationJsonSchemaCasesItemsGradersItems

- `kind` (string, required)

### V1EvalSuitesPostRequestBodyContentApplicationJsonSchemaCasesItemsInput

- `checkpoint` (V1EvalSuitesPostRequestBodyContentApplicationJsonSchemaCasesItemsInputCheckpoint, optional)
- `endUser` (V1EvalSuitesPostRequestBodyContentApplicationJsonSchemaCasesItemsInputEndUser, optional) — End user this case runs as (exposed to tools as \_endUser.\*, used to mint \{\{\_identity.token}}). Overrides the suite identityDefaults.endUser. Required (or a suite default) when the agent sets config.tenancyStrategy.
- `messages` (list of V1EvalSuitesPostRequestBodyContentApplicationJsonSchemaCasesItemsInputMessagesItems, optional)
- `record` (V1EvalSuitesPostRequestBodyContentApplicationJsonSchemaCasesItemsInputRecord, optional) — The record this case runs on, read by the flow as _record.name, _record.type and _record.metadata.*. Omitted fields keep the defaults: a record of type "eval" with empty metadata. Case variables still override same-named metadata fields at the top level.
- `tenant` (V1EvalSuitesPostRequestBodyContentApplicationJsonSchemaCasesItemsInputTenant, optional) — Product tenant this case runs within (exposed as _tenant.*). Overrides the suite identityDefaults.tenant.
- `toolMocks` (list of V1EvalSuitesPostRequestBodyContentApplicationJsonSchemaCasesItemsInputToolMocksItems, optional)
- `variables` (map from string to any, optional)

### V1EvalSuitesPostRequestBodyContentApplicationJsonSchemaIdentityDefaultsEndUser

The end user (the person) the turn runs as. Exposed to tools as _endUser.*, drives memory-profile sharding, and satisfies an end-user-isolated tenancyStrategy. Distinct from _user (the Builder User).

- `id` (string, required)
- `email` (string, optional)

### V1EvalSuitesPostRequestBodyContentApplicationJsonSchemaIdentityDefaultsTenant

The product tenant (a customer of your product, under your Builder Org) the turn runs within. Exposed to tools as _tenant.*.

- `id` (string, required)

### EvalSuiteDetailIdentityDefaultsEndUser

The end user (the person) the turn runs as. Exposed to tools as _endUser.*, drives memory-profile sharding, and satisfies an end-user-isolated tenancyStrategy. Distinct from _user (the Builder User).

- `id` (string, required)
- `email` (string, optional)

### EvalSuiteDetailIdentityDefaultsTenant

The product tenant (a customer of your product, under your Builder Org) the turn runs within. Exposed to tools as _tenant.*.

- `id` (string, required)

### V1EvalSuitesPostRequestBodyContentApplicationJsonSchemaCasesItemsInputCheckpoint

- `forkMessageIndex` (integer, required)
- `sourceConversationId` (string, optional)
- `sourceExecutionId` (string, optional)

### V1EvalSuitesPostRequestBodyContentApplicationJsonSchemaCasesItemsInputEndUser

End user this case runs as (exposed to tools as \_endUser.\*, used to mint \{\{\_identity.token}}). Overrides the suite identityDefaults.endUser. Required (or a suite default) when the agent sets config.tenancyStrategy.

- `id` (string, required)
- `email` (string, optional)

### V1EvalSuitesPostRequestBodyContentApplicationJsonSchemaCasesItemsInputMessagesItems

- `content` (string, required)
- `role` (string, required)
- `toolCalls` (list of V1EvalSuitesPostRequestBodyContentApplicationJsonSchemaCasesItemsInputMessagesItemsToolCallsItems, optional)
- `toolResults` (list of V1EvalSuitesPostRequestBodyContentApplicationJsonSchemaCasesItemsInputMessagesItemsToolResultsItems, optional)

### V1EvalSuitesPostRequestBodyContentApplicationJsonSchemaCasesItemsInputRecord

The record this case runs on, read by the flow as _record.name, _record.type and _record.metadata.*. Omitted fields keep the defaults: a record of type "eval" with empty metadata. Case variables still override same-named metadata fields at the top level.

- `metadata` (map from string to any, optional)
- `name` (string, optional)
- `type` (string, optional)

### V1EvalSuitesPostRequestBodyContentApplicationJsonSchemaCasesItemsInputTenant

Product tenant this case runs within (exposed as _tenant.*). Overrides the suite identityDefaults.tenant.

- `id` (string, required)

### V1EvalSuitesPostRequestBodyContentApplicationJsonSchemaCasesItemsInputToolMocksItems

- `output` (V1EvalSuitesPostRequestBodyContentApplicationJsonSchemaCasesItemsInputToolMocksItemsOutput, required, nullable)
- `toolName` (string, required)
- `input` (any, optional)
- `isError` (boolean, optional)
- `sourceToolExecutionId` (string, optional)
- `truncated` (boolean, optional)

### V1EvalSuitesPostRequestBodyContentApplicationJsonSchemaCasesItemsInputMessagesItemsToolCallsItems

- `toolCallId` (string, required)
- `toolName` (string, required)
- `args` (any, optional)

### V1EvalSuitesPostRequestBodyContentApplicationJsonSchemaCasesItemsInputMessagesItemsToolResultsItems

- `toolCallId` (string, required)
- `toolName` (string, required)
- `result` (any, optional)

### V1EvalSuitesPostRequestBodyContentApplicationJsonSchemaCasesItemsInputToolMocksItemsOutput

- boolean
- double
- string
- list of any
- map from string to any

## Examples

**Request**

```json
{
  "name": "string"
}
```

**Response**

```json
{
  "agentId": "string",
  "baselineBatchExecutionId": "string",
  "caseCount": 1,
  "cases": [
    {
      "createdAt": "string",
      "enabled": true,
      "graders": [
        null
      ],
      "id": "string",
      "name": "string",
      "notes": "string",
      "origin": "string",
      "suiteId": "string",
      "updatedAt": "string",
      "input": null
    }
  ],
  "createdAt": "string",
  "description": "string",
  "enabledCaseCount": 1,
  "flowId": "string",
  "graders": [
    null
  ],
  "id": "string",
  "identityDefaults": {
    "endUser": {
      "id": "string",
      "email": "string"
    },
    "tenant": {
      "id": "string"
    }
  },
  "isDefault": true,
  "judgeAgreement": {
    "agree": 1,
    "total": 1
  },
  "lastModifiedSource": "string",
  "latestRun": {
    "completedAt": "string",
    "passedCases": 1,
    "platformAbortedCases": 1,
    "runId": "string",
    "score": 1.1,
    "startedAt": "string",
    "status": "string",
    "totalCases": 1
  },
  "name": "string",
  "recordedToolMode": "next_step",
  "recordedToolUnmatchedPolicy": "fail",
  "targetName": "string",
  "targetType": "flow",
  "updatedAt": "string",
  "warnings": [
    {
      "code": "string",
      "message": "string",
      "agentId": "string"
    }
  ]
}
```

**SDK Code**

```python
import requests

url = "https://api.runtype.com/v1/eval/suites"

payload = { "name": "string" }
headers = {
    "Authorization": "Bearer <token>",
    "Content-Type": "application/json"
}

response = requests.post(url, json=payload, headers=headers)

print(response.json())
```

```javascript
const url = 'https://api.runtype.com/v1/eval/suites';
const options = {
  method: 'POST',
  headers: {Authorization: 'Bearer <token>', 'Content-Type': 'application/json'},
  body: '{"name":"string"}'
};

try {
  const response = await fetch(url, options);
  const data = await response.json();
  console.log(data);
} catch (error) {
  console.error(error);
}
```

```go
package main

import (
	"fmt"
	"strings"
	"net/http"
	"io"
)

func main() {

	url := "https://api.runtype.com/v1/eval/suites"

	payload := strings.NewReader("{\n  \"name\": \"string\"\n}")

	req, _ := http.NewRequest("POST", url, payload)

	req.Header.Add("Authorization", "Bearer <token>")
	req.Header.Add("Content-Type", "application/json")

	res, _ := http.DefaultClient.Do(req)

	defer res.Body.Close()
	body, _ := io.ReadAll(res.Body)

	fmt.Println(res)
	fmt.Println(string(body))

}
```

```ruby
require 'uri'
require 'net/http'

url = URI("https://api.runtype.com/v1/eval/suites")

http = Net::HTTP.new(url.host, url.port)
http.use_ssl = true

request = Net::HTTP::Post.new(url)
request["Authorization"] = 'Bearer <token>'
request["Content-Type"] = 'application/json'
request.body = "{\n  \"name\": \"string\"\n}"

response = http.request(request)
puts response.read_body
```

```java
import com.mashape.unirest.http.HttpResponse;
import com.mashape.unirest.http.Unirest;

HttpResponse<String> response = Unirest.post("https://api.runtype.com/v1/eval/suites")
  .header("Authorization", "Bearer <token>")
  .header("Content-Type", "application/json")
  .body("{\n  \"name\": \"string\"\n}")
  .asString();
```

```php
<?php
require_once('vendor/autoload.php');

$client = new \GuzzleHttp\Client();

$response = $client->request('POST', 'https://api.runtype.com/v1/eval/suites', [
  'body' => '{
  "name": "string"
}',
  'headers' => [
    'Authorization' => 'Bearer <token>',
    'Content-Type' => 'application/json',
  ],
]);

echo $response->getBody();
```

```csharp
using RestSharp;

var client = new RestClient("https://api.runtype.com/v1/eval/suites");
var request = new RestRequest(Method.POST);
request.AddHeader("Authorization", "Bearer <token>");
request.AddHeader("Content-Type", "application/json");
request.AddParameter("application/json", "{\n  \"name\": \"string\"\n}", ParameterType.RequestBody);
IRestResponse response = client.Execute(request);
```

```swift
import Foundation

let headers = [
  "Authorization": "Bearer <token>",
  "Content-Type": "application/json"
]
let parameters = ["name": "string"] as [String : Any]

let postData = JSONSerialization.data(withJSONObject: parameters, options: [])

let request = NSMutableURLRequest(url: NSURL(string: "https://api.runtype.com/v1/eval/suites")! as URL,
                                        cachePolicy: .useProtocolCachePolicy,
                                    timeoutInterval: 10.0)
request.httpMethod = "POST"
request.allHTTPHeaderFields = headers
request.httpBody = postData as Data

let session = URLSession.shared
let dataTask = session.dataTask(with: request as URLRequest, completionHandler: { (data, response, error) -> Void in
  if (error != nil) {
    print(error as Any)
  } else {
    let httpResponse = response as? HTTPURLResponse
    print(httpResponse)
  }
})

dataTask.resume()
```