> For clean Markdown of any page, append .md to the page URL. > For a complete documentation index, see https://docs.runtype.com/llms.txt. > For AI client integration (Claude Code, Cursor, etc.), connect to the MCP server at https://docs.runtype.com/_mcp/server. # Run evals from the API, MCP, and CLI Compare configurations, evaluate a Surface, and read suite definitions without the Flow editor. This page covers the eval tasks that [Running an eval](/user-guide/evals/running-an-eval) and [Manage evals as code](/user-guide/evals/manage-evals-as-code) don't: streaming one test input, submitting comparisons from an MCP client, evaluating a Surface, pulling a suite definition, and the `runtype eval` commands other than `run`. ## Before you begin You need the following: * A Runtype API key. Streaming and submitting evals require the `FLOWS:EXECUTE` scope. Pulling a suite definition requires `EVALS:READ` or `EVALS:WRITE`. For more information, see [Managing API keys](/user-guide/settings/managing-api-keys). * For the CLI tasks, the `runtype` CLI: `npm install -g @runtypelabs/cli`. * For the MCP tasks, an MCP client connected to the Runtype MCP server. ## Stream a comparison for one test input Send `POST /v1/eval/stream` to run one test input against one or more configurations at the same time and receive the results as Server-Sent Events (SSE). This is the API form of a Realtime eval in the dashboard. Runtype doesn't save the results. The following request compares two models on every prompt step of a Flow: ```bash curl -N https://api.runtype.com/v1/eval/stream \ -H "Authorization: Bearer YOUR_API_KEY" \ -H "Content-Type: application/json" \ -d '{ "flowId": "YOUR_FLOW_ID", "record": { "name": "double charge", "metadata": { "message": "I was double charged" } }, "evalConfigs": [ { "evalConfigId": "baseline", "evalName": "Baseline", "overrides": { "flow": { "model": "claude-sonnet-5" } } }, { "evalConfigId": "fast", "evalName": "Fast", "overrides": { "flow": { "model": "claude-haiku-4-5" } } } ] }' ``` Replace the following: * `YOUR_API_KEY`: your Runtype API key. * `YOUR_FLOW_ID`: the ID of the Flow to evaluate. The request body accepts the following fields: * Exactly one target: `flowId`, `flowDefinition` (an unsaved Flow), or `agentId`. `agentId` accepts only a Claude Managed Agent. For a standard Agent, use `POST /v1/eval/submit` or an eval suite. * At least one input: `record`, with the Record's fields under `metadata`, or `messages` for a conversation. * `evalConfigs`: one entry per configuration. `overrides.flow` applies to every prompt step. `overrides.steps` is keyed by step ID and takes precedence. Every event in the stream carries `executionId`, set to the `evalConfigId` of the configuration that produced it. Use that field to sort the interleaved events by configuration. Each configuration also emits `custom` events named `runtype.eval`, whose `value.phase` is `start`, `complete`, or `error`. Each request counts once against your daily eval limit. ## Submit a batch comparison from an MCP client The `submit_eval` tool of the Runtype MCP server queues an ungraded batch comparison. It returns the outputs of each configuration for you to compare, not pass or fail results. For graded results, use `create_eval_suite` and `run_eval_suite`. The following `submit_eval` arguments compare two models on an Agent against two Records: ```json { "agent_id": "YOUR_AGENT_ID", "record_ids": ["YOUR_RECORD_ID_1", "YOUR_RECORD_ID_2"], "eval_group_id": "support-model-choice", "eval_configs": [ { "evalName": "Baseline", "overrides": { "agent": { "model": "claude-sonnet-5" } } }, { "evalName": "Fast", "overrides": { "agent": { "model": "claude-haiku-4-5" } } } ] } ``` Replace the following: * `YOUR_AGENT_ID`: the ID of the Agent to evaluate. * `YOUR_RECORD_ID_1`, `YOUR_RECORD_ID_2`: the IDs of the Records to run. For an Agent, each Record carries its conversation in `metadata.messages`. Use `overrides.agent` on an Agent target and `overrides.flow` on a Flow target. Sending the wrong one returns an error. An Agent target requires `eval_configs`. For a Flow target, you can instead send `flow_id` with a single `eval_config`. External Agents aren't supported by this tool. To read the results, use the following tools: * `get_batch_status`: check whether a batch has finished. * `get_eval_results`: get the results of one batch. * `compare_eval`: compare aggregate metrics across the batches in the group. ## Evaluate a Surface A Surface eval runs a test input through a Surface and its connected Capabilities, so you can compare model and routing settings for the whole Product entry point. To evaluate a Surface, follow these steps: 1. Open the Product. 2. In Visual view, open the Surface's **More actions** menu and click **Evaluate**. In List view, the same menu item is **Run Eval**. The **Surface Eval** panel opens. 3. Select **Realtime** to run one test input immediately, or **Batch** to queue runs across all Records of one type. 4. For **Realtime**, enter **Messages** for a chat Surface, or JSON in **Test Input** for other Surfaces. For **Batch**, select a **Record Type** and, optionally, enter a **Group ID (optional)**. 5. Set up one to five configurations. For each configuration, select a model and, optionally, open **Surface Overrides** to change the **Router Model**, **Router Prompt**, **Orchestrator Capability**, or **Enabled Capabilities**. 6. Click **Run Eval**, **Compare N Configs**, or **Schedule Batch**. Realtime results appear in the panel and aren't saved. Batch runs appear on the **Runs** tab of the **Evals** page, grouped by their group ID. If the Surface has no connected Capabilities, the eval returns "No capabilities configured on this surface". ## Pull a suite definition Send `GET /v1/eval/pull` to get the saved definition of an eval suite by name. Use it when someone changed a suite in the dashboard and you want to copy the change into your repository. The following request pulls the suite named `smoke`: ```bash curl "https://api.runtype.com/v1/eval/pull?name=smoke" \ -H "Authorization: Bearer YOUR_API_KEY" ``` Replace `YOUR_API_KEY` with your Runtype API key. The response contains `suiteId`, `definition`, `contentHash`, `lastModifiedSource`, and `updatedAt`. The `definition` uses the same shape that `POST /v1/eval/ensure` accepts, with each case's full grader set under `expect`. A name that matches no suite returns HTTP `404`. The SDK equivalent is `Runtype.evals.pull('smoke')`. ## Use the CLI eval commands The `runtype eval` command group provides the following subcommands: | Command | Purpose | | ------------------------------------------------------- | ------------------------------------------------------------- | | `runtype eval run` | Runs `*.eval.ts` suites and grades them. Use it as a CI gate. | | `runtype eval submit --flow --records ` | Submits a Flow batch against the Record IDs in a JSON file. | | `runtype eval list` | Lists eval batches. Accepts `--limit`. | | `runtype eval results ` | Gets the results of one batch. | | `runtype eval compare ` | Requests a comparison of the batches in a group. | For `runtype eval run`, see [Run as a CI gate](/user-guide/evals/manage-evals-as-code#run-as-a-ci-gate). The `--records` file holds a JSON array of Record IDs, or an object with a `recordIds` array: ```json ["YOUR_RECORD_ID_1", "YOUR_RECORD_ID_2"] ``` Pass `--json` to `submit`, `list`, and `results`. Without it, these commands don't show the API response correctly: `submit` shows no batch ID, `list` reports no batches, and `results` shows no results. The `--json` output contains the full response. `submit` ignores `--name`, and `list` ignores `--flow`. `runtype eval compare` fails because the API rejects its request. To compare a group, send `POST /v1/eval/compare` with `evalGroupId` instead: ```bash curl -X POST https://api.runtype.com/v1/eval/compare \ -H "Authorization: Bearer YOUR_API_KEY" \ -H "Content-Type: application/json" \ -d '{"evalGroupId": "YOUR_EVAL_GROUP_ID"}' ``` Replace the following: * `YOUR_API_KEY`: your Runtype API key. * `YOUR_EVAL_GROUP_ID`: the ID of the eval group to compare. ## Troubleshooting The following table lists common errors and how to resolve them: | Error | Cause | Resolution | | --------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------- | | `Either record or messages is required` | A stream request has no test input. | Add `record` or `messages`. | | `AGENT_TYPE_EVAL_UNSUPPORTED` | A stream request targets an external Agent. | Use `POST /v1/eval/submit` for a baseline batch eval. | | HTTP `402` with `DAILY_EVAL_LIMIT` | The account used its daily eval limit. | Wait for the limit to reset at midnight UTC. | | HTTP `422` with `RUNTIME_LANE_INELIGIBLE` | A batch eval targets a Flow that batch can't run, such as a Flow with a top-level `wait-until` or `crawl` step, or an override makes the Flow unrunnable. Nothing is queued, and the eval doesn't count toward the daily limit. | Remove or replace the step or override that `reasons` names. `evalConfigIndex` names the configuration. | | HTTP `403` with `FLOWS:EXECUTE permission required` | The API key lacks the `FLOWS:EXECUTE` scope. | Use an API key with `FLOWS:EXECUTE`. | ## Next steps Continue with these guides: * [Running an eval](/user-guide/evals/running-an-eval): run Flow evals from the dashboard. * [Interpreting eval results](/user-guide/evals/interpreting-eval-results): compare metrics across a group. * [Manage evals as code](/user-guide/evals/manage-evals-as-code): define graded suites and run them in CI. * [Product views](/user-guide/products-surfaces/product-views): find a Surface's actions in Visual view and List view. * [Runtype MCP server](/developer-guides/integrations/runtype-mcp-server): connect an MCP client to Runtype.