> For clean Markdown of any page, append .md to the page URL.
> For a complete documentation index, see https://docs.runtype.com/llms.txt.
> For AI client integration (Claude Code, Cursor, etc.), connect to the MCP server at https://docs.runtype.com/_mcp/server.

# Compare evals

POST https://api.runtype.com/v1/eval/compare
Content-Type: application/json

Compare aggregate metrics across eval batches in a group.

Reference: https://docs.runtype.com/api-reference/runtype-api/evals/compare-evals

## Authentication

- `Authorization` header (bearer token, required) — API key or Clerk session token

## Servers

- `https://api.runtype.com` (Production, default)
- `https://api.runtype-staging.com` (Staging)

## Request

### Body (application/json)

This endpoint expects an object.

- `evalGroupId` (string, required)
- `stepIds` (list of string, optional)

## Response

### 200

Comparison returned

- `evalGroupId` (string, required)
- `evals` (list of V1EvalComparePostResponsesContentApplicationJsonSchemaEvalsItems, required)

## Errors

### 400 Bad Request Error

Invalid request

- `error` (string, required)
- `code` (string, optional) — Stable error code. `insufficient_scope` marks an API key that lacks a scope this request needs.
- `details` (list of ErrorDetailsItems, optional)
- `requiredScopes` (list of string, optional) — On `insufficient_scope`: every API key scope this request needs.

### 401 Unauthorized Error

Unauthorized

- `error` (string, required)
- `code` (string, optional) — Stable error code. `insufficient_scope` marks an API key that lacks a scope this request needs.
- `details` (list of ErrorDetailsItems, optional)
- `requiredScopes` (list of string, optional) — On `insufficient_scope`: every API key scope this request needs.

### 403 Forbidden Error

Forbidden

- `error` (string, required)
- `code` (string, optional) — Stable error code. `insufficient_scope` marks an API key that lacks a scope this request needs.
- `details` (list of ErrorDetailsItems, optional)
- `requiredScopes` (list of string, optional) — On `insufficient_scope`: every API key scope this request needs.

### 404 Not Found Error

Not found

- `error` (string, required)
- `code` (string, optional) — Stable error code. `insufficient_scope` marks an API key that lacks a scope this request needs.
- `details` (list of ErrorDetailsItems, optional)
- `requiredScopes` (list of string, optional) — On `insufficient_scope`: every API key scope this request needs.

### 500 Internal Server Error

Internal server error

- `code` (string, required) — Stable error code: `INTERNAL_ERROR` unless the route documents its own.
- `error` (string, required)
- `details` (list of InternalErrorDetailsItems, optional)
- `requiredScopes` (list of string, optional) — On `insufficient_scope`: every API key scope this request needs.
- `message` (string, optional) — Plain-language guidance: retry, then contact support with the `requestId`.
- `requestId` (string, optional) — The `X-Runtype-Request-ID` of the failed request, for support.

### 503 Service Unavailable Error

Database temporarily unavailable; nothing was read or written. Retry after the `Retry-After` interval (seconds).

- `code` (enum, required) — Stable error code. `insufficient_scope` marks an API key that lacks a scope this request needs.
  - Allowed values: `DATABASE_UNAVAILABLE`
- `error` (string, required)
- `details` (list of DatabaseUnavailableErrorDetailsItems, optional)
- `requiredScopes` (list of string, optional) — On `insufficient_scope`: every API key scope this request needs.
- `requestId` (string, optional) — The `X-Runtype-Request-ID` of the failed request, for support.

## Types

### V1EvalComparePostResponsesContentApplicationJsonSchemaEvalsItems

- `batchExecutionId` (string, required)
- `metrics` (V1EvalComparePostResponsesContentApplicationJsonSchemaEvalsItemsMetrics, required)
- `evalConfig` (any, optional)
- `evalName` (string, optional)

### ErrorDetailsItems

- `message` (string, required)
- `code` (string, optional) — Stable Runtype validation code (e.g. MISSING_REQUIRED_FIELD, INVALID_ENUM).
- `path` (string, optional) — Dotted JSON path to the offending field.
- `suggestion` (string, optional) — Actionable hint (allowed values, accepted range, …).

### InternalErrorDetailsItems

- `message` (string, required)
- `code` (string, optional) — Stable Runtype validation code (e.g. MISSING_REQUIRED_FIELD, INVALID_ENUM).
- `path` (string, optional) — Dotted JSON path to the offending field.
- `suggestion` (string, optional) — Actionable hint (allowed values, accepted range, …).

### DatabaseUnavailableErrorDetailsItems

- `message` (string, required)
- `code` (string, optional) — Stable Runtype validation code (e.g. MISSING_REQUIRED_FIELD, INVALID_ENUM).
- `path` (string, optional) — Dotted JSON path to the offending field.
- `suggestion` (string, optional) — Actionable hint (allowed values, accepted range, …).

### V1EvalComparePostResponsesContentApplicationJsonSchemaEvalsItemsMetrics

- `avgCostPerRecord` (double, required)
- `avgDurationPerRecord` (double, required)
- `failedSteps` (integer, required)
- `successRate` (double, required)
- `successfulSteps` (integer, required)
- `totalCompletionTokens` (integer, required)
- `totalCost` (double, required)
- `totalDurationMs` (double, required)
- `totalPromptTokens` (integer, required)
- `totalSteps` (integer, required)

## Examples

**Request**

```json
{
  "evalGroupId": "evalgrp_01jv0000000000000000000001",
  "stepIds": [
    "step_01jv0000000000000000000001",
    "step_01jv0000000000000000000002"
  ]
}
```

**Response**

```json
{
  "evalGroupId": "string",
  "evals": [
    {
      "batchExecutionId": "string",
      "metrics": {
        "avgCostPerRecord": 1.1,
        "avgDurationPerRecord": 1.1,
        "failedSteps": 1,
        "successRate": 1.1,
        "successfulSteps": 1,
        "totalCompletionTokens": 1,
        "totalCost": 1.1,
        "totalDurationMs": 1.1,
        "totalPromptTokens": 1,
        "totalSteps": 1
      },
      "evalName": "string"
    }
  ]
}
```

**SDK Code**

```python Compare Evals in a Group
import requests

url = "https://api.runtype.com/v1/eval/compare"

payload = {
    "evalGroupId": "evalgrp_01jv0000000000000000000001",
    "stepIds": ["step_01jv0000000000000000000001", "step_01jv0000000000000000000002"]
}
headers = {
    "Authorization": "Bearer <token>",
    "Content-Type": "application/json"
}

response = requests.post(url, json=payload, headers=headers)

print(response.json())
```

```javascript Compare Evals in a Group
const url = 'https://api.runtype.com/v1/eval/compare';
const options = {
  method: 'POST',
  headers: {Authorization: 'Bearer <token>', 'Content-Type': 'application/json'},
  body: '{"evalGroupId":"evalgrp_01jv0000000000000000000001","stepIds":["step_01jv0000000000000000000001","step_01jv0000000000000000000002"]}'
};

try {
  const response = await fetch(url, options);
  const data = await response.json();
  console.log(data);
} catch (error) {
  console.error(error);
}
```

```go Compare Evals in a Group
package main

import (
	"fmt"
	"strings"
	"net/http"
	"io"
)

func main() {

	url := "https://api.runtype.com/v1/eval/compare"

	payload := strings.NewReader("{\n  \"evalGroupId\": \"evalgrp_01jv0000000000000000000001\",\n  \"stepIds\": [\n    \"step_01jv0000000000000000000001\",\n    \"step_01jv0000000000000000000002\"\n  ]\n}")

	req, _ := http.NewRequest("POST", url, payload)

	req.Header.Add("Authorization", "Bearer <token>")
	req.Header.Add("Content-Type", "application/json")

	res, _ := http.DefaultClient.Do(req)

	defer res.Body.Close()
	body, _ := io.ReadAll(res.Body)

	fmt.Println(res)
	fmt.Println(string(body))

}
```

```ruby Compare Evals in a Group
require 'uri'
require 'net/http'

url = URI("https://api.runtype.com/v1/eval/compare")

http = Net::HTTP.new(url.host, url.port)
http.use_ssl = true

request = Net::HTTP::Post.new(url)
request["Authorization"] = 'Bearer <token>'
request["Content-Type"] = 'application/json'
request.body = "{\n  \"evalGroupId\": \"evalgrp_01jv0000000000000000000001\",\n  \"stepIds\": [\n    \"step_01jv0000000000000000000001\",\n    \"step_01jv0000000000000000000002\"\n  ]\n}"

response = http.request(request)
puts response.read_body
```

```java Compare Evals in a Group
import com.mashape.unirest.http.HttpResponse;
import com.mashape.unirest.http.Unirest;

HttpResponse<String> response = Unirest.post("https://api.runtype.com/v1/eval/compare")
  .header("Authorization", "Bearer <token>")
  .header("Content-Type", "application/json")
  .body("{\n  \"evalGroupId\": \"evalgrp_01jv0000000000000000000001\",\n  \"stepIds\": [\n    \"step_01jv0000000000000000000001\",\n    \"step_01jv0000000000000000000002\"\n  ]\n}")
  .asString();
```

```php Compare Evals in a Group
<?php
require_once('vendor/autoload.php');

$client = new \GuzzleHttp\Client();

$response = $client->request('POST', 'https://api.runtype.com/v1/eval/compare', [
  'body' => '{
  "evalGroupId": "evalgrp_01jv0000000000000000000001",
  "stepIds": [
    "step_01jv0000000000000000000001",
    "step_01jv0000000000000000000002"
  ]
}',
  'headers' => [
    'Authorization' => 'Bearer <token>',
    'Content-Type' => 'application/json',
  ],
]);

echo $response->getBody();
```

```csharp Compare Evals in a Group
using RestSharp;

var client = new RestClient("https://api.runtype.com/v1/eval/compare");
var request = new RestRequest(Method.POST);
request.AddHeader("Authorization", "Bearer <token>");
request.AddHeader("Content-Type", "application/json");
request.AddParameter("application/json", "{\n  \"evalGroupId\": \"evalgrp_01jv0000000000000000000001\",\n  \"stepIds\": [\n    \"step_01jv0000000000000000000001\",\n    \"step_01jv0000000000000000000002\"\n  ]\n}", ParameterType.RequestBody);
IRestResponse response = client.Execute(request);
```

```swift Compare Evals in a Group
import Foundation

let headers = [
  "Authorization": "Bearer <token>",
  "Content-Type": "application/json"
]
let parameters = [
  "evalGroupId": "evalgrp_01jv0000000000000000000001",
  "stepIds": ["step_01jv0000000000000000000001", "step_01jv0000000000000000000002"]
] as [String : Any]

let postData = JSONSerialization.data(withJSONObject: parameters, options: [])

let request = NSMutableURLRequest(url: NSURL(string: "https://api.runtype.com/v1/eval/compare")! as URL,
                                        cachePolicy: .useProtocolCachePolicy,
                                    timeoutInterval: 10.0)
request.httpMethod = "POST"
request.allHTTPHeaderFields = headers
request.httpBody = postData as Data

let session = URLSession.shared
let dataTask = session.dataTask(with: request as URLRequest, completionHandler: { (data, response, error) -> Void in
  if (error != nil) {
    print(error as Any)
  } else {
    let httpResponse = response as? HTTPURLResponse
    print(httpResponse)
  }
})

dataTask.resume()
```