> For clean Markdown of any page, append .md to the page URL.
> For a complete documentation index, see https://docs.runtype.com/llms.txt.
> For AI client integration (Claude Code, Cursor, etc.), connect to the MCP server at https://docs.runtype.com/_mcp/server.

# Ensure eval suite (config-as-code converge)

POST https://api.runtype.com/v1/eval/ensure
Content-Type: application/json

Idempotently converge a repo-defined eval suite (target + cases + graders) onto the platform. Identity is the suite name + account scope; the target flow/agent is resolved by name. Hash-only probes skip the payload in the steady state (a miss is a normal 200 `definitionRequired` response, not an error); full requests create the suite or replace the cases ensure owns when the canonical content hash differs. Cases saved from runs, added in the dashboard, or owned by a product are never replaced. Set dryRun to plan without writing (CI drift gate), and onConflict: "overwrite" to converge over dashboard/API edits. Non-executing: never runs the eval. The response always carries the server-computed canonical hash.

Reference: https://docs.runtype.com/api-reference/runtype-api/evals/ensure-eval-suite-config-as-code-converge

## Authentication

- `Authorization` header (bearer token, required) — API key or Clerk session token

## Servers

- `https://api.runtype.com` (Production, default)
- `https://api.runtype-staging.com` (Staging)

## Request

### Body (application/json)

This endpoint expects an object.

- `name` (string, required)
- `contentHash` (string, optional)
- `definition` (V1EvalEnsurePostRequestBodyContentApplicationJsonSchemaDefinition, optional)
- `dryRun` (boolean, optional)
- `expectedRemoteHash` (string, optional)
- `onConflict` (enum, optional)
  - Allowed values: `error`, `overwrite`

## Response

### 200

Converge result: unchanged | created | updated | definitionRequired (hash miss — retry with the full definition) | plan (dryRun)

- `EvalEnsureResponse`
  - EvalEnsureResponse0
  - EvalEnsureResponse1
  - EvalEnsureResponse2

## Errors

### 400 Bad Request Error

Validation error (bad definition, unresolved target, divergent per-case graders, or virtual)

- `error` (string, required)
- `code` (string, optional) — Stable error code. `insufficient_scope` marks an API key that lacks a scope this request needs.
- `details` (list of ErrorDetailsItems, optional)
- `requiredScopes` (list of string, optional) — On `insufficient_scope`: every API key scope this request needs.

### 401 Unauthorized Error

Unauthorized

- `error` (string, required)
- `code` (string, optional) — Stable error code. `insufficient_scope` marks an API key that lacks a scope this request needs.
- `details` (list of ErrorDetailsItems, optional)
- `requiredScopes` (list of string, optional) — On `insufficient_scope`: every API key scope this request needs.

### 403 Forbidden Error

Insufficient permissions

- `error` (string, required)
- `code` (string, optional) — Stable error code. `insufficient_scope` marks an API key that lacks a scope this request needs.
- `details` (list of ErrorDetailsItems, optional)
- `requiredScopes` (list of string, optional) — On `insufficient_scope`: every API key scope this request needs.

### 409 Conflict Error

Conflict: external_modification (last write was not ensure; pass onConflict: "overwrite") or remote_changed (expectedRemoteHash guard)

- `code` (enum, required)
  - Allowed values: `external_modification`, `remote_changed`
- `error` (string, required)
- `currentHash` (string, optional, nullable)
- `lastModifiedSource` (string, optional)
- `modifiedAt` (string, optional, nullable)

### 422 Unprocessable Entity Error

Submitted contentHash does not match the server-recomputed canonical hash

- `error` (string, required)
- `code` (string, optional) — Stable error code. `insufficient_scope` marks an API key that lacks a scope this request needs.
- `details` (list of ErrorDetailsItems, optional)
- `requiredScopes` (list of string, optional) — On `insufficient_scope`: every API key scope this request needs.

### 500 Internal Server Error

Internal server error

- `error` (string, required)
- `code` (string, optional) — Stable error code. `insufficient_scope` marks an API key that lacks a scope this request needs.
- `details` (list of ErrorDetailsItems, optional)
- `requiredScopes` (list of string, optional) — On `insufficient_scope`: every API key scope this request needs.

## Types

### V1EvalEnsurePostRequestBodyContentApplicationJsonSchemaDefinition

- `cases` (list of V1EvalEnsurePostRequestBodyContentApplicationJsonSchemaDefinitionCasesItems, required)
- `name` (string, required)
- `target` (V1EvalEnsurePostRequestBodyContentApplicationJsonSchemaDefinitionTarget, required)
- `identityDefaults` (V1EvalEnsurePostRequestBodyContentApplicationJsonSchemaDefinitionIdentityDefaults, optional) — Default identity (endUser / tenant) every case runs as unless the case sets its own. Omitting it on a converge clears the stored default.
- `recordedToolMode` (enum, optional) — How cases carrying toolMocks replay — captured (saved_from_run) or hand-authored alike: "next_step" (default; captured cases only) captures the first tool intent and stops, or "continue" serves each case's toolMocks by tool name and runs past the first action.
  - Allowed values: `next_step`, `continue`
- `recordedToolUnmatchedPolicy` (enum, optional) — For "continue"-mode replays, the unmatched-call policy: "fail" (default) or "stub".
  - Allowed values: `fail`, `stub`
- `virtual` (boolean, optional)

### EvalEnsureResponse0

- `contentHash` (string, required) — Server-computed canonical content hash. Clients should echo this hash in probes.
- `result` (enum, required)
  - Allowed values: `unchanged`, `created`, `updated`
- `suiteId` (string, required)

### EvalEnsureResponse1

- `result` (enum, required)
  - Allowed values: `definitionRequired`

### EvalEnsureResponse2

- `changedKeys` (list of string, required)
- `changes` (enum, required)
  - Allowed values: `none`, `create`, `update`
- `contentHash` (string, required)
- `result` (enum, required)
  - Allowed values: `plan`
- `remoteHash` (string, optional)
- `suiteId` (string, optional)

### ErrorDetailsItems

- `message` (string, required)
- `code` (string, optional) — Stable Runtype validation code (e.g. MISSING_REQUIRED_FIELD, INVALID_ENUM).
- `path` (string, optional) — Dotted JSON path to the offending field.
- `suggestion` (string, optional) — Actionable hint (allowed values, accepted range, …).

### V1EvalEnsurePostRequestBodyContentApplicationJsonSchemaDefinitionCasesItems

- `expect` (list of V1EvalEnsurePostRequestBodyContentApplicationJsonSchemaDefinitionCasesItemsExpectItems, required)
- `name` (string, required)
- `expected` (V1EvalEnsurePostRequestBodyContentApplicationJsonSchemaDefinitionCasesItemsExpected, optional)
- `input` (V1EvalEnsurePostRequestBodyContentApplicationJsonSchemaDefinitionCasesItemsInput, optional, default: {})

### V1EvalEnsurePostRequestBodyContentApplicationJsonSchemaDefinitionTarget

- `agent` (string, optional)
- `flow` (string, optional)

### V1EvalEnsurePostRequestBodyContentApplicationJsonSchemaDefinitionIdentityDefaults

Default identity (endUser / tenant) every case runs as unless the case sets its own. Omitting it on a converge clears the stored default.

- `endUser` (V1EvalEnsurePostRequestBodyContentApplicationJsonSchemaDefinitionIdentityDefaultsEndUser, optional) — The end user (the person) the turn runs as. Exposed to tools as _endUser.*, drives memory-profile sharding, and satisfies an end-user-isolated tenancyStrategy. Distinct from _user (the Builder User).
- `tenant` (V1EvalEnsurePostRequestBodyContentApplicationJsonSchemaDefinitionIdentityDefaultsTenant, optional) — The product tenant (a customer of your product, under your Builder Org) the turn runs within. Exposed to tools as _tenant.*.

### V1EvalEnsurePostRequestBodyContentApplicationJsonSchemaDefinitionCasesItemsExpectItems

- `kind` (string, required)

### V1EvalEnsurePostRequestBodyContentApplicationJsonSchemaDefinitionCasesItemsExpected

- `facts` (list of string, optional)
- `json` (any, optional)
- `text` (string, optional)

### V1EvalEnsurePostRequestBodyContentApplicationJsonSchemaDefinitionCasesItemsInput

- `endUser` (V1EvalEnsurePostRequestBodyContentApplicationJsonSchemaDefinitionCasesItemsInputEndUser, optional) — The end user (the person) the turn runs as. Exposed to tools as _endUser.*, drives memory-profile sharding, and satisfies an end-user-isolated tenancyStrategy. Distinct from _user (the Builder User).
- `messages` (list of V1EvalEnsurePostRequestBodyContentApplicationJsonSchemaDefinitionCasesItemsInputMessagesItems, optional)
- `tenant` (V1EvalEnsurePostRequestBodyContentApplicationJsonSchemaDefinitionCasesItemsInputTenant, optional) — The product tenant (a customer of your product, under your Builder Org) the turn runs within. Exposed to tools as _tenant.*.
- `toolMocks` (list of V1EvalEnsurePostRequestBodyContentApplicationJsonSchemaDefinitionCasesItemsInputToolMocksItems, optional)
- `variables` (map from string to any, optional)

### V1EvalEnsurePostRequestBodyContentApplicationJsonSchemaDefinitionIdentityDefaultsEndUser

The end user (the person) the turn runs as. Exposed to tools as _endUser.*, drives memory-profile sharding, and satisfies an end-user-isolated tenancyStrategy. Distinct from _user (the Builder User).

- `id` (string, required)
- `email` (string, optional)

### V1EvalEnsurePostRequestBodyContentApplicationJsonSchemaDefinitionIdentityDefaultsTenant

The product tenant (a customer of your product, under your Builder Org) the turn runs within. Exposed to tools as _tenant.*.

- `id` (string, required)

### V1EvalEnsurePostRequestBodyContentApplicationJsonSchemaDefinitionCasesItemsInputEndUser

The end user (the person) the turn runs as. Exposed to tools as _endUser.*, drives memory-profile sharding, and satisfies an end-user-isolated tenancyStrategy. Distinct from _user (the Builder User).

- `id` (string, required)
- `email` (string, optional)

### V1EvalEnsurePostRequestBodyContentApplicationJsonSchemaDefinitionCasesItemsInputMessagesItems

- `content` (string, required)
- `role` (string, required)

### V1EvalEnsurePostRequestBodyContentApplicationJsonSchemaDefinitionCasesItemsInputTenant

The product tenant (a customer of your product, under your Builder Org) the turn runs within. Exposed to tools as _tenant.*.

- `id` (string, required)

### V1EvalEnsurePostRequestBodyContentApplicationJsonSchemaDefinitionCasesItemsInputToolMocksItems

- `output` (V1EvalEnsurePostRequestBodyContentApplicationJsonSchemaDefinitionCasesItemsInputToolMocksItemsOutput, required, nullable)
- `toolName` (string, required)
- `input` (any, optional)
- `isError` (boolean, optional)
- `sourceToolExecutionId` (string, optional)
- `truncated` (boolean, optional)

### V1EvalEnsurePostRequestBodyContentApplicationJsonSchemaDefinitionCasesItemsInputToolMocksItemsOutput

- boolean
- double
- string
- list of any
- map from string to any

## Examples

**Request**

```json
{
  "name": "string"
}
```

**Response**

```json
{
  "contentHash": "string",
  "result": "unchanged",
  "suiteId": "string"
}
```

**SDK Code**

```python
import requests

url = "https://api.runtype.com/v1/eval/ensure"

payload = { "name": "string" }
headers = {
    "Authorization": "Bearer <token>",
    "Content-Type": "application/json"
}

response = requests.post(url, json=payload, headers=headers)

print(response.json())
```

```javascript
const url = 'https://api.runtype.com/v1/eval/ensure';
const options = {
  method: 'POST',
  headers: {Authorization: 'Bearer <token>', 'Content-Type': 'application/json'},
  body: '{"name":"string"}'
};

try {
  const response = await fetch(url, options);
  const data = await response.json();
  console.log(data);
} catch (error) {
  console.error(error);
}
```

```go
package main

import (
	"fmt"
	"strings"
	"net/http"
	"io"
)

func main() {

	url := "https://api.runtype.com/v1/eval/ensure"

	payload := strings.NewReader("{\n  \"name\": \"string\"\n}")

	req, _ := http.NewRequest("POST", url, payload)

	req.Header.Add("Authorization", "Bearer <token>")
	req.Header.Add("Content-Type", "application/json")

	res, _ := http.DefaultClient.Do(req)

	defer res.Body.Close()
	body, _ := io.ReadAll(res.Body)

	fmt.Println(res)
	fmt.Println(string(body))

}
```

```ruby
require 'uri'
require 'net/http'

url = URI("https://api.runtype.com/v1/eval/ensure")

http = Net::HTTP.new(url.host, url.port)
http.use_ssl = true

request = Net::HTTP::Post.new(url)
request["Authorization"] = 'Bearer <token>'
request["Content-Type"] = 'application/json'
request.body = "{\n  \"name\": \"string\"\n}"

response = http.request(request)
puts response.read_body
```

```java
import com.mashape.unirest.http.HttpResponse;
import com.mashape.unirest.http.Unirest;

HttpResponse<String> response = Unirest.post("https://api.runtype.com/v1/eval/ensure")
  .header("Authorization", "Bearer <token>")
  .header("Content-Type", "application/json")
  .body("{\n  \"name\": \"string\"\n}")
  .asString();
```

```php
<?php
require_once('vendor/autoload.php');

$client = new \GuzzleHttp\Client();

$response = $client->request('POST', 'https://api.runtype.com/v1/eval/ensure', [
  'body' => '{
  "name": "string"
}',
  'headers' => [
    'Authorization' => 'Bearer <token>',
    'Content-Type' => 'application/json',
  ],
]);

echo $response->getBody();
```

```csharp
using RestSharp;

var client = new RestClient("https://api.runtype.com/v1/eval/ensure");
var request = new RestRequest(Method.POST);
request.AddHeader("Authorization", "Bearer <token>");
request.AddHeader("Content-Type", "application/json");
request.AddParameter("application/json", "{\n  \"name\": \"string\"\n}", ParameterType.RequestBody);
IRestResponse response = client.Execute(request);
```

```swift
import Foundation

let headers = [
  "Authorization": "Bearer <token>",
  "Content-Type": "application/json"
]
let parameters = ["name": "string"] as [String : Any]

let postData = JSONSerialization.data(withJSONObject: parameters, options: [])

let request = NSMutableURLRequest(url: NSURL(string: "https://api.runtype.com/v1/eval/ensure")! as URL,
                                        cachePolicy: .useProtocolCachePolicy,
                                    timeoutInterval: 10.0)
request.httpMethod = "POST"
request.allHTTPHeaderFields = headers
request.httpBody = postData as Data

let session = URLSession.shared
let dataTask = session.dataTask(with: request as URLRequest, completionHandler: { (data, response, error) -> Void in
  if (error != nil) {
    print(error as Any)
  } else {
    let httpResponse = response as? HTTPURLResponse
    print(httpResponse)
  }
})

dataTask.resume()
```