Compare evals | Runtype Documentation

curl -X POST https://api.runtype.com/v1/eval/compare \
     -H "Authorization: <apiKey>" \
     -H "Content-Type: application/json" \
     -d '{
  "evalGroupId": "evalgrp_01jv0000000000000000000001",
  "stepIds": [
    "step_01jv0000000000000000000001",
    "step_01jv0000000000000000000002"
  ]
}'

{
  "evalGroupId": "string",
  "evals": [
    {
      "batchExecutionId": "string",
      "metrics": {
        "avgCostPerRecord": 1.1,
        "avgDurationPerRecord": 1.1,
        "failedSteps": 1,
        "successRate": 1.1,
        "successfulSteps": 1,
        "totalCompletionTokens": 1,
        "totalCost": 1.1,
        "totalDurationMs": 1.1,
        "totalPromptTokens": 1,
        "totalSteps": 1
      },
      "evalName": "string"
    }
  ]
}

Compare aggregate metrics across eval batches in a group.

Authentication

AuthorizationBearer

API key or Clerk session token

Request

This endpoint expects an object.

evalGroupIdstringRequired

stepIdslist of stringsOptional

Response

Comparison returned

evalGroupIdstring

evalslist of objects

Errors

400

Bad Request Error

401

Unauthorized Error

403

Forbidden Error

404

Not Found Error

500

Internal Server Error