> For clean Markdown of any page, append .md to the page URL.
> For a complete documentation index, see https://docs.runtype.com/llms.txt.
> For AI client integration (Claude Code, Cursor, etc.), connect to the MCP server at https://docs.runtype.com/_mcp/server.

# Pull eval suite definition (config-as-code)

GET https://api.runtype.com/v1/eval/pull

Return the canonical config-as-code definition and provenance for an eval suite by name — the absorb-drift direction of the ensure protocol. The definition carries the cases ensure owns (or, for a suite ensure never wrote, its unowned cases), and the contentHash is recomputed from that live state.

Reference: https://docs.runtype.com/api-reference/runtype-api/evals/pull-eval-suite-definition-config-as-code

## Authentication

- `Authorization` header (bearer token, required) — API key or Clerk session token

## Servers

- `https://api.runtype.com` (Production, default)
- `https://api.runtype-staging.com` (Staging)

## Request

### Query parameters

- `name` (string, required)

## Response

### 200

Canonical definition + provenance

- `contentHash` (string, required)
- `definition` (EvalPullResponseDefinition, required)
- `lastModifiedSource` (string, required, nullable)
- `suiteId` (string, required)
- `updatedAt` (string, required, nullable)

## Errors

### 401 Unauthorized Error

Unauthorized

- `error` (string, required)
- `code` (string, optional) — Stable error code. `insufficient_scope` marks an API key that lacks a scope this request needs.
- `details` (list of ErrorDetailsItems, optional)
- `requiredScopes` (list of string, optional) — On `insufficient_scope`: every API key scope this request needs.

### 403 Forbidden Error

Insufficient permissions

- `error` (string, required)
- `code` (string, optional) — Stable error code. `insufficient_scope` marks an API key that lacks a scope this request needs.
- `details` (list of ErrorDetailsItems, optional)
- `requiredScopes` (list of string, optional) — On `insufficient_scope`: every API key scope this request needs.

### 404 Not Found Error

No eval suite with that name in the account scope

- `error` (string, required)
- `code` (string, optional) — Stable error code. `insufficient_scope` marks an API key that lacks a scope this request needs.
- `details` (list of ErrorDetailsItems, optional)
- `requiredScopes` (list of string, optional) — On `insufficient_scope`: every API key scope this request needs.

### 500 Internal Server Error

Internal server error

- `error` (string, required)
- `code` (string, optional) — Stable error code. `insufficient_scope` marks an API key that lacks a scope this request needs.
- `details` (list of ErrorDetailsItems, optional)
- `requiredScopes` (list of string, optional) — On `insufficient_scope`: every API key scope this request needs.

## Types

### EvalPullResponseDefinition

- `cases` (list of EvalPullResponseDefinitionCasesItems, required)
- `name` (string, required)
- `target` (EvalPullResponseDefinitionTarget, required)
- `identityDefaults` (EvalPullResponseDefinitionIdentityDefaults, optional) — Default identity (endUser / tenant) every case runs as unless the case sets its own. Omitting it on a converge clears the stored default.
- `recordedToolMode` (enum, optional) — How cases carrying toolMocks replay — captured (saved_from_run) or hand-authored alike: "next_step" (default; captured cases only) captures the first tool intent and stops, or "continue" serves each case's toolMocks by tool name and runs past the first action.
  - Allowed values: `next_step`, `continue`
- `recordedToolUnmatchedPolicy` (enum, optional) — For "continue"-mode replays, the unmatched-call policy: "fail" (default) or "stub".
  - Allowed values: `fail`, `stub`
- `virtual` (boolean, optional)

### ErrorDetailsItems

- `message` (string, required)
- `code` (string, optional) — Stable Runtype validation code (e.g. MISSING_REQUIRED_FIELD, INVALID_ENUM).
- `path` (string, optional) — Dotted JSON path to the offending field.
- `suggestion` (string, optional) — Actionable hint (allowed values, accepted range, …).

### EvalPullResponseDefinitionCasesItems

- `expect` (list of EvalPullResponseDefinitionCasesItemsExpectItems, required)
- `name` (string, required)
- `expected` (EvalPullResponseDefinitionCasesItemsExpected, optional)
- `input` (EvalPullResponseDefinitionCasesItemsInput, optional, default: {})

### EvalPullResponseDefinitionTarget

- `agent` (string, optional)
- `flow` (string, optional)

### EvalPullResponseDefinitionIdentityDefaults

Default identity (endUser / tenant) every case runs as unless the case sets its own. Omitting it on a converge clears the stored default.

- `endUser` (EvalPullResponseDefinitionIdentityDefaultsEndUser, optional) — The end user (the person) the turn runs as. Exposed to tools as _endUser.*, drives memory-profile sharding, and satisfies an end-user-isolated tenancyStrategy. Distinct from _user (the Builder User).
- `tenant` (EvalPullResponseDefinitionIdentityDefaultsTenant, optional) — The product tenant (a customer of your product, under your Builder Org) the turn runs within. Exposed to tools as _tenant.*.

### EvalPullResponseDefinitionCasesItemsExpectItems

- `kind` (string, required)

### EvalPullResponseDefinitionCasesItemsExpected

- `facts` (list of string, optional)
- `json` (any, optional)
- `text` (string, optional)

### EvalPullResponseDefinitionCasesItemsInput

- `endUser` (EvalPullResponseDefinitionCasesItemsInputEndUser, optional) — The end user (the person) the turn runs as. Exposed to tools as _endUser.*, drives memory-profile sharding, and satisfies an end-user-isolated tenancyStrategy. Distinct from _user (the Builder User).
- `messages` (list of EvalPullResponseDefinitionCasesItemsInputMessagesItems, optional)
- `tenant` (EvalPullResponseDefinitionCasesItemsInputTenant, optional) — The product tenant (a customer of your product, under your Builder Org) the turn runs within. Exposed to tools as _tenant.*.
- `toolMocks` (list of EvalPullResponseDefinitionCasesItemsInputToolMocksItems, optional)
- `variables` (map from string to any, optional)

### EvalPullResponseDefinitionIdentityDefaultsEndUser

The end user (the person) the turn runs as. Exposed to tools as _endUser.*, drives memory-profile sharding, and satisfies an end-user-isolated tenancyStrategy. Distinct from _user (the Builder User).

- `id` (string, required)
- `email` (string, optional)

### EvalPullResponseDefinitionIdentityDefaultsTenant

The product tenant (a customer of your product, under your Builder Org) the turn runs within. Exposed to tools as _tenant.*.

- `id` (string, required)

### EvalPullResponseDefinitionCasesItemsInputEndUser

The end user (the person) the turn runs as. Exposed to tools as _endUser.*, drives memory-profile sharding, and satisfies an end-user-isolated tenancyStrategy. Distinct from _user (the Builder User).

- `id` (string, required)
- `email` (string, optional)

### EvalPullResponseDefinitionCasesItemsInputMessagesItems

- `content` (string, required)
- `role` (string, required)

### EvalPullResponseDefinitionCasesItemsInputTenant

The product tenant (a customer of your product, under your Builder Org) the turn runs within. Exposed to tools as _tenant.*.

- `id` (string, required)

### EvalPullResponseDefinitionCasesItemsInputToolMocksItems

- `output` (EvalPullResponseDefinitionCasesItemsInputToolMocksItemsOutput, required, nullable)
- `toolName` (string, required)
- `input` (any, optional)
- `isError` (boolean, optional)
- `sourceToolExecutionId` (string, optional)
- `truncated` (boolean, optional)

### EvalPullResponseDefinitionCasesItemsInputToolMocksItemsOutput

- boolean
- double
- string
- list of any
- map from string to any

## Examples

**Response**

```json
{
  "contentHash": "string",
  "definition": {
    "cases": [
      {
        "expect": [
          {
            "kind": "string"
          }
        ],
        "name": "string",
        "expected": {
          "facts": [
            "string"
          ],
          "text": "string"
        },
        "input": {
          "endUser": {
            "id": "string",
            "email": "string"
          },
          "messages": [
            {
              "content": "string",
              "role": "string"
            }
          ],
          "tenant": {
            "id": "string"
          },
          "toolMocks": [
            {
              "output": {},
              "toolName": "string",
              "isError": true,
              "sourceToolExecutionId": "string",
              "truncated": true
            }
          ],
          "variables": {}
        }
      }
    ],
    "name": "string",
    "target": {
      "agent": "string",
      "flow": "string"
    },
    "identityDefaults": {
      "endUser": {
        "id": "string",
        "email": "string"
      },
      "tenant": {
        "id": "string"
      }
    },
    "recordedToolMode": "next_step",
    "recordedToolUnmatchedPolicy": "fail",
    "virtual": true
  },
  "lastModifiedSource": "string",
  "suiteId": "string",
  "updatedAt": "string"
}
```

**SDK Code**

```python
import requests

url = "https://api.runtype.com/v1/eval/pull"

querystring = {"name":"name"}

headers = {"Authorization": "Bearer <token>"}

response = requests.get(url, headers=headers, params=querystring)

print(response.json())
```

```javascript
const url = 'https://api.runtype.com/v1/eval/pull?name=name';
const options = {method: 'GET', headers: {Authorization: 'Bearer <token>'}};

try {
  const response = await fetch(url, options);
  const data = await response.json();
  console.log(data);
} catch (error) {
  console.error(error);
}
```

```go
package main

import (
	"fmt"
	"net/http"
	"io"
)

func main() {

	url := "https://api.runtype.com/v1/eval/pull?name=name"

	req, _ := http.NewRequest("GET", url, nil)

	req.Header.Add("Authorization", "Bearer <token>")

	res, _ := http.DefaultClient.Do(req)

	defer res.Body.Close()
	body, _ := io.ReadAll(res.Body)

	fmt.Println(res)
	fmt.Println(string(body))

}
```

```ruby
require 'uri'
require 'net/http'

url = URI("https://api.runtype.com/v1/eval/pull?name=name")

http = Net::HTTP.new(url.host, url.port)
http.use_ssl = true

request = Net::HTTP::Get.new(url)
request["Authorization"] = 'Bearer <token>'

response = http.request(request)
puts response.read_body
```

```java
import com.mashape.unirest.http.HttpResponse;
import com.mashape.unirest.http.Unirest;

HttpResponse<String> response = Unirest.get("https://api.runtype.com/v1/eval/pull?name=name")
  .header("Authorization", "Bearer <token>")
  .asString();
```

```php
<?php
require_once('vendor/autoload.php');

$client = new \GuzzleHttp\Client();

$response = $client->request('GET', 'https://api.runtype.com/v1/eval/pull?name=name', [
  'headers' => [
    'Authorization' => 'Bearer <token>',
  ],
]);

echo $response->getBody();
```

```csharp
using RestSharp;

var client = new RestClient("https://api.runtype.com/v1/eval/pull?name=name");
var request = new RestRequest(Method.GET);
request.AddHeader("Authorization", "Bearer <token>");
IRestResponse response = client.Execute(request);
```

```swift
import Foundation

let headers = ["Authorization": "Bearer <token>"]

let request = NSMutableURLRequest(url: NSURL(string: "https://api.runtype.com/v1/eval/pull?name=name")! as URL,
                                        cachePolicy: .useProtocolCachePolicy,
                                    timeoutInterval: 10.0)
request.httpMethod = "GET"
request.allHTTPHeaderFields = headers

let session = URLSession.shared
let dataTask = session.dataTask(with: request as URLRequest, completionHandler: { (data, response, error) -> Void in
  if (error != nil) {
    print(error as Any)
  } else {
    let httpResponse = response as? HTTPURLResponse
    print(httpResponse)
  }
})

dataTask.resume()
```