> For clean Markdown of any page, append .md to the page URL. > For a complete documentation index, see https://docs.runtype.com/llms.txt. > For AI client integration (Claude Code, Cursor, etc.), connect to the MCP server at https://docs.runtype.com/_mcp/server. # Improving your agents from production traces > Use production traces to find failures, capture eval cases, change an agent, and verify the result Use production traces to improve an agent. Read a real execution, capture a failure as an eval case, change the agent, and run the suite to check the result. Runtype records Flow and Agent executions in Logs. A trace includes agent iterations, model calls, tool invocations, flow steps, duration, token usage, and cost. You don't add instrumentation to inspect executions that Runtype runs. Agents that run elsewhere join the same improvement loop by sending their traces to Runtype. For details, see [Agents that run outside Runtype](#agents-that-run-outside-runtype). Evals support deterministic checks, agent trace checks, and AI graders, so you can turn a production failure into a regression case. You can capture only Agent executions as eval cases. **Add to eval** doesn't appear for Flow executions. This guide uses the following improvement loop: 1. **Observe.** Find an execution that failed or returned a wrong answer. 2. **Diagnose.** Read the trace and locate the iteration, tool call, or model turn that failed. 3. **Capture.** Save the execution as an eval case before changing the agent. 4. **Fix.** Change the agent's instructions, tools, or model. 5. **Verify.** Run the suite. Confirm that captured and existing cases pass. Run the improvement loop from the dashboard, from a coding agent through the Runtype Model Context Protocol (MCP) server, or from your own tooling through the REST API. ## Before you begin You need the following: * A Runtype Agent with at least one recorded execution. * For the MCP server, an MCP client such as Claude Code or Cursor. * For the REST API, an API key with the `AGENTS:READ` and `EVALS:WRITE` permissions. For details, see [Authentication](/developer-guides/getting-started/authentication). ## Why start with production failures Use these practices when you build or maintain an eval suite: * **Start with real failures.** Use production outputs to identify failure modes instead of inventing cases from a brainstorm. Name and count each mode. * **Capture before changing the agent.** A captured case records the failure before a prompt, tool, or model change. Use the case to check whether the change fixes the failure. * **Record intended behavior.** Write the correct behavior in the expected output or grader criteria. Do not copy the incorrect output. Exclude provider outages and other service failures from regression cases. End-user feedback can also propose cases. For a Flow target that has a suite, a thumbs-down or a low NPS or CSAT rating adds a proposal to the suite's review queue with a **From feedback** badge. For details, see [Managing eval suites](/user-guide/evals/managing-eval-suites#review-proposed-cases). ## Use the dashboard The following procedures run the improvement loop in the dashboard. ### Find the run To find a failed execution, follow these steps: 1. Open **Logs**. 2. Use the **Status**, **Type**, and **Time range** controls to narrow the list. Presets range from 15 minutes to 30 days. To set a specific interval, choose **Custom range**. You can also select a range in the timeline chart or turn on **Live**. 3. Select a row with **Failed** status. The detail panel opens. ### Inspect the trace The detail panel shows the execution timeline, including iterations, tool calls, and model turns. For a run that belongs to a conversation, the panel has three tabs. The **Conversation** tab shows the transcript, **Timeline** shows the execution tree, and **Together** shows both side by side. Select a node to see its input, output, and errors in the inspector. Look for the following signals: * **Failed nodes**: Find the node with an error status and inspect its input and error message. * **Tool calls**: Compare tool names and arguments with the intended behavior. * **Context per iteration**: Review estimated input tokens across iterations. When the model context size is known, the chart marks its ceiling. ### Capture the case To capture an Agent execution as an eval case, follow these steps: 1. In the inspector, click **Add to eval**. The **Add to eval** dialog opens. 2. In the **Eval** list, select an eval. If the Agent has no default eval, select **Create default eval for AGENT_NAME**, where AGENT_NAME is the Agent's name. Runtype creates the eval when you save the case. 3. If the run includes tool calls, use **Fork point** to select the action that the eval grades next. If the run has no tool calls, the eval grades the agent's reply. 4. Click **Add to eval**. The dialog reports whether the case is fully or partially replayable. The case freezes the conversation at the fork point and grades the agent's next step. Recorded tool results are stored as editable mocks. If a tool output is too large, Runtype truncates it at capture and the case is partially replayable. During a suite run, the case replays the truncated tool results and still grades the next step. ### Fix and verify What a save changes, and what the suite grades, depends on your organization's live-default setting: * With the setting off, a save changes what production traffic runs, and **Run** grades the saved configuration. * With the setting on, a save doesn't change production, and **Run** grades the version that the `live` alias serves, not your saved change. The dashboard **Run** button doesn't take a version. To grade a saved version before it goes live, run the suite from the MCP server, the REST API, or the CLI with a version or alias selector. For details, see [Agent versioning and publishing](/user-guide/agents/agent-versioning-and-publishing#evaluate-a-version-before-deploying-it). To fix and verify, follow these steps: 1. In the Agent editor, change the agent's instructions, tools, or model. 2. Click **Save version**. 3. On the eval suite page, click **Run**. 4. Confirm that the captured case passes and existing cases do not regress. 5. When an AI grader cites evidence, inspect its reasoning and the cited evidence. 6. Optional: Use the thumbs-up or thumbs-down control to record whether the verdict matches your judgment. The **Judge trust** stat shows the number of agreement reviews and total reviews. ## Use the MCP server Use the Runtype MCP server to run the improvement loop from Claude Code, Cursor, or another MCP client. The loop uses the following tools: * **Find**: `list_logs`, or `list_runs` with `agent_id` set to the agent's ID and `status` set to `"failed"`. Both return the last 24 hours by default. To search further back, pass `from` with an ISO 8601 timestamp. `get_log_stats` returns aggregate log counts. * **Diagnose**: `trace_execution` or `trace_conversation`. * **Capture**: `get_eval_capture_preview` to choose a fork point, then `add_eval_case_from_execution`. * **Fix**: `update_agent`, which saves the change as a draft version. `list_agent_versions` returns the version's ID. * **Verify**: `run_eval_suite` with `agent_version_id` or `agent_alias` to grade that version. A suite with more than 50 enabled cases runs as a queued run, so read its scores with `get_eval_run_scores` after it completes. After the suite passes, deploy with `publish_agent_version`. To configure Claude Code manually, run the following command: **`Claude Code`** ```bash title="Claude Code" claude mcp add --transport http runtype https://api.runtype.com/v1/mcp/protocol ``` For an all-in-one setup, run `npx -y @runtypelabs/cli@latest onboard`. It configures the MCP connection and installs Runtype agent skills. For details, see [Connect Runtype to coding agents](/developer-guides/integrations/connect-coding-agents). Give your coding agent a prompt like the following: **`Improvement-loop prompt`** ```plaintext title="Improvement-loop prompt" My Runtype agent "AGENT_NAME" has returned wrong answers. Use the Runtype MCP tools to run an improvement loop: 1. Find the failure. Call `list_runs` with `agent_id` set to the agent's ID and `status` set to `"failed"`. Pass `from` if the failure is older than 24 hours. Call `list_logs` with `level` set to `"error"` and `order` set to `"desc"` for error details. 2. Diagnose it. Call `trace_execution` and inspect the tree. Identify the iteration, tool call, or model turn that failed. Quote trace evidence before you draw a conclusion. 3. Capture it. Call `get_eval_capture_preview` to review fork points. Call `add_eval_case_from_execution` to save the execution in the agent's eval suite. If no suite exists, call `create_eval_suite` first. Write the expected behavior, not the observed incorrect output. 4. Fix it. Propose a change to the agent's instructions or tool configuration. Show me the change. Apply it with `update_agent` after I approve it, then call `list_agent_versions` to find the draft version it saved. 5. Verify it. Call `run_eval_suite` with `agent_version_id` set to that version. If the run is queued, poll `get_eval_run_scores`. Confirm that the captured case passes and existing cases do not regress. If a case fails, revise the change and run the suite again. Do not publish the version until I approve it. After each run, report what you found, what you changed, and the suite score before and after. ``` Replace `AGENT_NAME` with your Runtype Agent name. Repeat this sequence after each production incident. Each captured case adds a concrete failure to the suite and gives later changes a case to check. ## Use the REST API Use the following REST endpoints to automate the improvement loop from your own tooling: | Step | Endpoint | | --------------------------- | ----------------------------------------------------------------------------------------- | | Find failing executions | `GET /v1/logs?level=error&order=desc`, `GET /v1/runs?agentId=YOUR_AGENT_ID&status=failed` | | Aggregate log counts | `GET /v1/logs/stats` | | Pull a trace tree | `GET /v1/logs/trace/execution/YOUR_EXECUTION_ID` | | Explain a run with AI | `POST /v1/logs/trace/execution/YOUR_EXECUTION_ID/explain` | | Trace a conversation | `GET /v1/logs/trace/conversation/YOUR_CONVERSATION_ID` | | Preview capture fork points | `GET /v1/eval/executions/YOUR_EXECUTION_ID/capture-preview` | | Capture an eval case | `POST /v1/eval/suites/YOUR_SUITE_ID/cases/from-execution` | | Run the suite | `POST /v1/eval/suites/YOUR_SUITE_ID/run` | | Get run scores | `GET /v1/eval/runs/YOUR_RUN_ID/scores` | Replace `YOUR_AGENT_ID`, `YOUR_EXECUTION_ID`, `YOUR_CONVERSATION_ID`, `YOUR_SUITE_ID`, and `YOUR_RUN_ID` with IDs from your account. Send your API key as a bearer token with each request. The logs and runs endpoints return the last 24 hours by default. To search further back, pass `from` with an ISO 8601 timestamp. A trace response contains a hierarchical tree of `iteration`, `model`, `tool`, and `step` nodes. It also returns aggregate `totalDuration`, `totalTokens`, and `totalCost` values. The explain endpoint returns observations, possible explanations, and next steps whose `eventIds` match entry `id` values in that trace tree. It runs a metered model call in your account, counts as one execution, and needs the `ANALYTICS:READ` and `FLOWS:EXECUTE` scopes for an API key. To capture a case, preview the execution first. The preview's `actions` array lists each recorded tool call with a `forkMessageIndex`. The following request previews an execution: **`cURL`** ```bash title="cURL" curl https://api.runtype.com/v1/eval/executions/YOUR_EXECUTION_ID/capture-preview \ -H "Authorization: Bearer YOUR_API_KEY" ``` The following request captures the execution, forks it before the chosen action, and records the intended behavior: **`cURL`** ```bash title="cURL" curl -X POST https://api.runtype.com/v1/eval/suites/YOUR_SUITE_ID/cases/from-execution \ -H "Authorization: Bearer YOUR_API_KEY" \ -H "Content-Type: application/json" \ -d '{ "executionId": "YOUR_EXECUTION_ID", "forkMessageIndex": 4, "expected": { "text": "Look up the order before issuing a refund." } }' ``` Replace the following: * `YOUR_API_KEY`: your Runtype API key * `YOUR_EXECUTION_ID`: the ID of the Agent execution to capture * `YOUR_SUITE_ID`: the eval suite that receives the case * `4`: a `forkMessageIndex` value from the preview. If you omit it, the fork is immediately before the last tool call. For a run with no tool calls, the case grades the agent's reply. The request also accepts optional `name` and `notes` fields. The response's `replayable` field is `false` when some tool outputs were truncated at capture. A suite with 50 or fewer enabled cases runs inline and returns its scores. A larger suite returns `202` with a `runId`. Read its scores from `GET /v1/eval/runs/YOUR_RUN_ID/scores` after it completes. To grade a saved version before it goes live, send `{"agent": {"versionId": "YOUR_VERSION_ID"}}` or `{"agent": {"alias": "YOUR_ALIAS"}}` in the run request body. The TypeScript SDK exposes the same calls as `Runtype.evals.suites.getCapturePreview`, `addCaseFromExecution`, and `run`. For request and response shapes, see the [API reference](/api-reference). ## Troubleshooting The following problems are the most common: * **`403`**: The API key lacks a permission. Reading Agent logs and runs needs `AGENTS:READ`. Capture needs `EVALS:WRITE`. The capture preview needs `EVALS:READ`, `EVALS:WRITE`, or `AGENTS:READ`. Running a suite needs `FLOWS:EXECUTE` or `EVALS:WRITE`. * **`404` with "not found or not accessible" on capture or preview**: The execution is not an Agent execution, or it belongs to another account or organization. * **No failures in the list**: The logs and runs endpoints return the last 24 hours unless you pass `from`. Runs older than your plan's log retention period are not returned. For details, see [Log retention](/user-guide/logs/what-are-logs#log-retention). * **`402` on a suite run**: The run exceeded a limit. With `code: "DAILY_EVAL_LIMIT"`, retry after the daily eval limit resets at midnight UTC. With `code: "SPEND_LIMIT_EXCEEDED"`, the organization reached its spend limit. ## Agents that run outside Runtype An agent built with Flue, the Vercel AI SDK, LangGraph, the OpenAI Agents SDK, Mastra, OpenClaw, Hermes, Pi, or a custom agent loop joins the improvement loop when it exports OpenTelemetry traces to Runtype. To set up the export, see [Bring your own agent](/developer-guides/guides/bring-your-own-agent). For Flue on Cloudflare Workers, the export is a dashboard destination plus one `wrangler.jsonc` block, with no code. For the Cloudflare Agents SDK, read [its status](/developer-guides/guides/reporting-external-telemetry#cloudflare-agents-sdk) first. The native path needs `wrapAISDK`, with `storeMessages` and `storeTools` set so that the run has a transcript to capture. Cloudflare can group several turns into one trace or split one request across several traces, and Runtype creates one run per trace. The improvement loop differs in three places: * **Observe.** Ingested runs show an **External** badge in the **Runs** view and carry `otel-...` execution IDs. Their timings and cost are what the agent reported, not what Runtype measured or billed. * **Capture.** Capture needs the run's transcript. The exported spans must include `gen_ai.input.messages` and `gen_ai.output.messages`. Cloudflare Workers traces, `@flue/opentelemetry`, and `@runtypelabs/flue-otel` 0.5 or later include them by default. If you set `content: false` on `@runtypelabs/flue-otel`, the spans carry no transcript and you can't capture the run. The OpenClaw and Hermes plugins and the Pi coding agent extension send content only when you set `RUNTYPE_CAPTURE_CONTENT=true` (OpenClaw and Pi) or `RUNTYPE_OTEL_CAPTURE_CONTENT=true` (Hermes). The Pi durable binding sends content unless you pass `content: false`. * **Fix and verify.** You fix the agent in your repository, not with `update_agent`. `run_eval_suite` works only when Runtype can call the agent. An `external` agent registered with an Agent2Agent (A2A) endpoint qualifies, but the suite skips cases that replay recorded tool results. For an agent that only sends telemetry or uses a `runtype-stream` endpoint, re-run the scenario against your redeployed agent. Then inspect the new run with `trace_execution`. The captured cases remain the regression record. Give your coding agent a prompt like the following for an agent that runs outside Runtype: **`Improvement-loop prompt (external agent)`** ```plaintext title="Improvement-loop prompt (external agent)" My agent "AGENT_NAME" runs outside Runtype and sends its traces to Runtype. It has returned wrong answers. Use the Runtype MCP tools to run an improvement loop: 1. Find the failure. Call `list_runs` with `agent_id` set to the agent's ID and `status` set to `"failed"`. Pass `from` if the failure is older than 24 hours. 2. Diagnose it. Call `trace_execution` and inspect the tree. Identify the iteration, tool call, or model turn that failed. Quote trace evidence before you draw a conclusion. 3. Capture it. Call `get_eval_capture_preview` to review fork points. Call `add_eval_case_from_execution` to save the execution in the agent's eval suite. If no suite exists, call `create_eval_suite` first. Write the expected behavior, not the observed incorrect output. 4. Fix it. Propose the change to my agent's code and show me the diff. 5. Verify it. Tell me which scenario to re-run, then inspect the new run with `trace_execution`. After each run, report what you found and what you changed. ``` Replace `AGENT_NAME` with the name of the Runtype Agent that owns the ingested runs. ## Next steps Continue the improvement loop with these guides: * [Agent versioning and publishing](/user-guide/agents/agent-versioning-and-publishing): grade a version before it goes live * [Running an eval](/user-guide/evals/running-an-eval): choose an execution mode and review results * [Interpreting eval results](/user-guide/evals/interpreting-eval-results): read pass and fail outcomes and grader evidence * [Managing eval suites](/user-guide/evals/managing-eval-suites): manage cases, graders, runs, and captured-case replay * [Designing evals that work](/user-guide/evals/designing-evals-that-work): identify failure modes and write graders * [Working with Logs](/user-guide/logs/working-with-logs): inspect execution details, traces, and filters * [Manage evals as code](/user-guide/evals/manage-evals-as-code): run suites from your repository and CI * [Reporting telemetry from an external agent](/developer-guides/guides/reporting-external-telemetry): send OpenTelemetry traces from your own infrastructure * [Runtype MCP server](/developer-guides/integrations/runtype-mcp-server): review the tools and connection options * [Braintrust (tracing)](/user-guide/integrations/braintrust-tracing): send execution spans to your observability workspace * [Querying logs programmatically](/user-guide/logs/querying-logs-programmatically): query logs with the API, `runtype logs`, and `runtype tail` * [Working with runs](/user-guide/logs/working-with-runs): find runs and read the run detail page