Run evals from the API, MCP, and CLI

Compare configurations, evaluate a Surface, and read suite definitions without the Flow editor. This page covers the eval tasks that Running an eval and Manage evals as code don’t: streaming one test input, submitting comparisons from an MCP client, evaluating a Surface, pulling a suite definition, and the runtype eval commands other than run.

Before you begin

You need the following:

A Runtype API key. Streaming and submitting evals require the FLOWS:EXECUTE scope. Pulling a suite definition requires EVALS:READ or EVALS:WRITE. For more information, see Managing API keys.
For the CLI tasks, the runtype CLI: npm install -g @runtypelabs/cli.
For the MCP tasks, an MCP client connected to the Runtype MCP server.

Stream a comparison for one test input

Send POST /v1/eval/stream to run one test input against one or more configurations at the same time and receive the results as Server-Sent Events (SSE). This is the API form of a Realtime eval in the dashboard. Runtype doesn’t save the results.

The following request compares two models on every prompt step of a Flow:

curl -N https://api.runtype.com/v1/eval/stream \
  -H "Authorization: Bearer YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "flowId": "YOUR_FLOW_ID",
    "record": {
      "name": "double charge",
      "metadata": { "message": "I was double charged" }
    },
    "evalConfigs": [
      {
        "evalConfigId": "baseline",
        "evalName": "Baseline",
        "overrides": { "flow": { "model": "claude-sonnet-5" } }
      },
      {
        "evalConfigId": "fast",
        "evalName": "Fast",
        "overrides": { "flow": { "model": "claude-haiku-4-5" } }
      }
    ]
  }'

Replace the following:

YOUR_API_KEY: your Runtype API key.
YOUR_FLOW_ID: the ID of the Flow to evaluate.

The request body accepts the following fields:

Exactly one target: flowId, flowDefinition (an unsaved Flow), or agentId. agentId accepts only a Claude Managed Agent. For a standard Agent, use POST /v1/eval/submit or an eval suite.
At least one input: record, with the Record’s fields under metadata, or messages for a conversation.
evalConfigs: one entry per configuration. overrides.flow applies to every prompt step. overrides.steps is keyed by step ID and takes precedence.

Every event in the stream carries executionId, set to the evalConfigId of the configuration that produced it. Use that field to sort the interleaved events by configuration. Each configuration also emits custom events named runtype.eval, whose value.phase is start, complete, or error.

Each request counts once against your daily eval limit.

Submit a batch comparison from an MCP client

The submit_eval tool of the Runtype MCP server queues an ungraded batch comparison. It returns the outputs of each configuration for you to compare, not pass or fail results. For graded results, use create_eval_suite and run_eval_suite.

The following submit_eval arguments compare two models on an Agent against two Records:

{
  "agent_id": "YOUR_AGENT_ID",
  "record_ids": ["YOUR_RECORD_ID_1", "YOUR_RECORD_ID_2"],
  "eval_group_id": "support-model-choice",
  "eval_configs": [
    { "evalName": "Baseline", "overrides": { "agent": { "model": "claude-sonnet-5" } } },
    { "evalName": "Fast", "overrides": { "agent": { "model": "claude-haiku-4-5" } } }
  ]
}

Replace the following:

YOUR_AGENT_ID: the ID of the Agent to evaluate.
YOUR_RECORD_ID_1, YOUR_RECORD_ID_2: the IDs of the Records to run. For an Agent, each Record carries its conversation in metadata.messages.

Use overrides.agent on an Agent target and overrides.flow on a Flow target. Sending the wrong one returns an error. An Agent target requires eval_configs. For a Flow target, you can instead send flow_id with a single eval_config. External Agents aren’t supported by this tool.

To read the results, use the following tools:

get_batch_status: check whether a batch has finished.
get_eval_results: get the results of one batch.
compare_eval: compare aggregate metrics across the batches in the group.

Evaluate a Surface

A Surface eval runs a test input through a Surface and its connected Capabilities, so you can compare model and routing settings for the whole Product entry point.

To evaluate a Surface, follow these steps:

Open the Product.
In Visual view, open the Surface’s More actions menu and click Evaluate. In List view, the same menu item is Run Eval. The Surface Eval panel opens.
Select Realtime to run one test input immediately, or Batch to queue runs across all Records of one type.
For Realtime, enter Messages for a chat Surface, or JSON in Test Input for other Surfaces. For Batch, select a Record Type and, optionally, enter a Group ID (optional).
Set up one to five configurations. For each configuration, select a model and, optionally, open Surface Overrides to change the Router Model, Router Prompt, Orchestrator Capability, or Enabled Capabilities.
Click Run Eval, Compare N Configs, or Schedule Batch.

Realtime results appear in the panel and aren’t saved. Batch runs appear on the Runs tab of the Evals page, grouped by their group ID. If the Surface has no connected Capabilities, the eval returns “No capabilities configured on this surface”.

Pull a suite definition

Send GET /v1/eval/pull to get the saved definition of an eval suite by name. Use it when someone changed a suite in the dashboard and you want to copy the change into your repository.

The following request pulls the suite named smoke:

curl "https://api.runtype.com/v1/eval/pull?name=smoke" \
  -H "Authorization: Bearer YOUR_API_KEY"

Replace YOUR_API_KEY with your Runtype API key.

The response contains suiteId, definition, contentHash, lastModifiedSource, and updatedAt. The definition uses the same shape that POST /v1/eval/ensure accepts, with each case’s full grader set under expect. A name that matches no suite returns HTTP 404. The SDK equivalent is Runtype.evals.pull('smoke').

Use the CLI eval commands

The runtype eval command group provides the following subcommands:

Command	Purpose
`runtype eval run`	Runs `*.eval.ts` suites and grades them. Use it as a CI gate.
`runtype eval submit --flow <flow-id> --records <file>`	Submits a Flow batch against the Record IDs in a JSON file.
`runtype eval list`	Lists eval batches. Accepts `--limit`.
`runtype eval results <batch-id>`	Gets the results of one batch.
`runtype eval compare <group-id>`	Requests a comparison of the batches in a group.

For runtype eval run, see Run as a CI gate.

The --records file holds a JSON array of Record IDs, or an object with a recordIds array:

["YOUR_RECORD_ID_1", "YOUR_RECORD_ID_2"]

Pass --json to submit, list, and results. Without it, these commands don’t show the API response correctly: submit shows no batch ID, list reports no batches, and results shows no results. The --json output contains the full response. submit ignores --name, and list ignores --flow.

runtype eval compare fails because the API rejects its request. To compare a group, send POST /v1/eval/compare with evalGroupId instead:

curl -X POST https://api.runtype.com/v1/eval/compare \
  -H "Authorization: Bearer YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"evalGroupId": "YOUR_EVAL_GROUP_ID"}'

Replace the following:

YOUR_API_KEY: your Runtype API key.
YOUR_EVAL_GROUP_ID: the ID of the eval group to compare.

Troubleshooting

The following table lists common errors and how to resolve them:

Error	Cause	Resolution
`Either record or messages is required`	A stream request has no test input.	Add `record` or `messages`.
`AGENT_TYPE_EVAL_UNSUPPORTED`	A stream request targets an external Agent.	Use `POST /v1/eval/submit` for a baseline batch eval.
HTTP `402` with `DAILY_EVAL_LIMIT`	The account used its daily eval limit.	Wait for the limit to reset at midnight UTC.
HTTP `422` with `RUNTIME_LANE_INELIGIBLE`	A batch eval targets a Flow that batch can’t run, such as a Flow with a top-level `wait-until` or `crawl` step, or an override makes the Flow unrunnable. Nothing is queued, and the eval doesn’t count toward the daily limit.	Remove or replace the step or override that `reasons` names. `evalConfigIndex` names the configuration.
HTTP `403` with `FLOWS:EXECUTE permission required`	The API key lacks the `FLOWS:EXECUTE` scope.	Use an API key with `FLOWS:EXECUTE`.

Next steps

Continue with these guides:

Running an eval: run Flow evals from the dashboard.
Interpreting eval results: compare metrics across a group.
Manage evals as code: define graded suites and run them in CI.
Product views: find a Surface’s actions in Visual view and List view.
Runtype MCP server: connect an MCP client to Runtype.