Improving your agents from production traces

Use production traces to improve an agent. Read a real execution, capture a failure as an eval case, change the agent, and run the suite to check the result.

Runtype records Flow and Agent executions in Logs. A trace includes agent iterations, model calls, tool invocations, flow steps, duration, token usage, and cost. You don’t add instrumentation to inspect executions that Runtype runs. Agents that run elsewhere join the same improvement loop by sending their traces to Runtype. For details, see Agents that run outside Runtype. Evals support deterministic checks, agent trace checks, and AI graders, so you can turn a production failure into a regression case.

You can capture only Agent executions as eval cases. Add to eval doesn’t appear for Flow executions.

This guide uses the following improvement loop:

1. Observe. Find an execution that failed or returned a wrong answer.
2. Diagnose. Read the trace and locate the iteration, tool call, or model turn that failed.
3. Capture. Save the execution as an eval case before changing the agent.
4. Fix. Change the agent’s instructions, tools, or model.
5. Verify. Run the suite. Confirm that captured and existing cases pass.

Run the improvement loop from the dashboard, from a coding agent through the Runtype Model Context Protocol (MCP) server, or from your own tooling through the REST API.

Before you begin

You need the following:

• A Runtype Agent with at least one recorded execution.
• For the MCP server, an MCP client such as Claude Code or Cursor.
• For the REST API, an API key with the AGENTS:READ and EVALS:WRITE permissions. For details, see Authentication.

Why start with production failures

Use these practices when you build or maintain an eval suite:

• Start with real failures. Use production outputs to identify failure modes instead of inventing cases from a brainstorm. Name and count each mode.
• Capture before changing the agent. A captured case records the failure before a prompt, tool, or model change. Use the case to check whether the change fixes the failure.
• Record intended behavior. Write the correct behavior in the expected output or grader criteria. Do not copy the incorrect output. Exclude provider outages and other service failures from regression cases.

End-user feedback can also propose cases. For a Flow target that has a suite, a thumbs-down or a low NPS or CSAT rating adds a proposal to the suite’s review queue with a From feedback badge. For details, see Managing eval suites.

Use the dashboard

The following procedures run the improvement loop in the dashboard.

Find the run

To find a failed execution, follow these steps:

1. Open Logs.
2. Use the Status, Type, and Time range controls to narrow the list. Presets range from 15 minutes to 30 days. To set a specific interval, choose Custom range. You can also select a range in the timeline chart or turn on Live.
3. Select a row with Failed status. The detail panel opens.

Inspect the trace

The detail panel shows the execution timeline, including iterations, tool calls, and model turns. For a run that belongs to a conversation, the panel has three tabs. The Conversation tab shows the transcript, Timeline shows the execution tree, and Together shows both side by side. Select a node to see its input, output, and errors in the inspector.

Look for the following signals:

• Failed nodes: Find the node with an error status and inspect its input and error message.
• Tool calls: Compare tool names and arguments with the intended behavior.
• Context per iteration: Review estimated input tokens across iterations. When the model context size is known, the chart marks its ceiling.

Capture the case

To capture an Agent execution as an eval case, follow these steps:

1. In the inspector, click Add to eval. The Add to eval dialog opens.
2. In the Eval list, select an eval. If the Agent has no default eval, select Create default eval for AGENT_NAME, where AGENT_NAME is the Agent’s name. Runtype creates the eval when you save the case.
3. If the run includes tool calls, use Fork point to select the action that the eval grades next. If the run has no tool calls, the eval grades the agent’s reply.
4. Click Add to eval. The dialog reports whether the case is fully or partially replayable.

The case freezes the conversation at the fork point and grades the agent’s next step. Recorded tool results are stored as editable mocks. If a tool output is too large, Runtype truncates it at capture and the case is partially replayable. During a suite run, the case replays the truncated tool results and still grades the next step.

Fix and verify

What a save changes, and what the suite grades, depends on your organization’s live-default setting:

• With the setting off, a save changes what production traffic runs, and Run grades the saved configuration.
• With the setting on, a save doesn’t change production, and Run grades the version that the live alias serves, not your saved change.

The dashboard Run button doesn’t take a version. To grade a saved version before it goes live, run the suite from the MCP server, the REST API, or the CLI with a version or alias selector. For details, see Agent versioning and publishing.

To fix and verify, follow these steps:

1. In the Agent editor, change the agent’s instructions, tools, or model.
2. Click Save version.
3. On the eval suite page, click Run.
4. Confirm that the captured case passes and existing cases do not regress.
5. When an AI grader cites evidence, inspect its reasoning and the cited evidence.
6. Optional: Use the thumbs-up or thumbs-down control to record whether the verdict matches your judgment. The Judge trust stat shows the number of agreement reviews and total reviews.

Use the MCP server

Use the Runtype MCP server to run the improvement loop from Claude Code, Cursor, or another MCP client. The loop uses the following tools:

• Find: list_logs, or list_runs with agent_id set to the agent’s ID and status set to "failed". Both return the last 24 hours by default. To search further back, pass from with an ISO 8601 timestamp. get_log_stats returns aggregate log counts.
• Diagnose: trace_execution or trace_conversation.
• Capture: get_eval_capture_preview to choose a fork point, then add_eval_case_from_execution.
• Fix: update_agent, which saves the change as a draft version. list_agent_versions returns the version’s ID.
• Verify: run_eval_suite with agent_version_id or agent_alias to grade that version. A suite with more than 50 enabled cases runs as a queued run, so read its scores with get_eval_run_scores after it completes. After the suite passes, deploy with publish_agent_version.

To configure Claude Code manually, run the following command:

claude mcp add --transport http runtype https://api.runtype.com/v1/mcp/protocol

For an all-in-one setup, run npx -y @runtypelabs/cli@latest onboard. It configures the MCP connection and installs Runtype agent skills. For details, see Connect Runtype to coding agents.

Give your coding agent a prompt like the following:

My Runtype agent "AGENT_NAME" has returned wrong answers. Use the
Runtype MCP tools to run an improvement loop:
1. Find the failure. Call `list_runs` with `agent_id` set to the agent's ID
   and `status` set to `"failed"`. Pass `from` if the failure is older than
   24 hours. Call `list_logs` with `level` set to `"error"` and `order` set
   to `"desc"` for error details.
2. Diagnose it. Call `trace_execution` and inspect the tree. Identify the
   iteration, tool call, or model turn that failed. Quote trace evidence before
   you draw a conclusion.
3. Capture it. Call `get_eval_capture_preview` to review fork points. Call
   `add_eval_case_from_execution` to save the execution in the agent's eval
   suite. If no suite exists, call `create_eval_suite` first. Write the
   expected behavior, not the observed incorrect output.
4. Fix it. Propose a change to the agent's instructions or tool configuration.
   Show me the change. Apply it with `update_agent` after I approve it, then
   call `list_agent_versions` to find the draft version it saved.
5. Verify it. Call `run_eval_suite` with `agent_version_id` set to that
   version. If the run is queued, poll `get_eval_run_scores`. Confirm that the
   captured case passes and existing cases do not regress. If a case fails,
   revise the change and run the suite again. Do not publish the version
   until I approve it.
After each run, report what you found, what you changed, and the suite score
before and after.

Replace AGENT_NAME with your Runtype Agent name.

Repeat this sequence after each production incident. Each captured case adds a concrete failure to the suite and gives later changes a case to check.

Use the REST API

Use the following REST endpoints to automate the improvement loop from your own tooling:

Replace YOUR_AGENT_ID, YOUR_EXECUTION_ID, YOUR_CONVERSATION_ID, YOUR_SUITE_ID, and YOUR_RUN_ID with IDs from your account. Send your API key as a bearer token with each request.

The logs and runs endpoints return the last 24 hours by default. To search further back, pass from with an ISO 8601 timestamp. A trace response contains a hierarchical tree of iteration, model, tool, and step nodes. It also returns aggregate totalDuration, totalTokens, and totalCost values. The explain endpoint returns observations, possible explanations, and next steps whose eventIds match entry id values in that trace tree. It runs a metered model call in your account, counts as one execution, and needs the ANALYTICS:READ and FLOWS:EXECUTE scopes for an API key.

To capture a case, preview the execution first. The preview’s actions array lists each recorded tool call with a forkMessageIndex. The following request previews an execution:

curl https://api.runtype.com/v1/eval/executions/YOUR_EXECUTION_ID/capture-preview \
  -H "Authorization: Bearer YOUR_API_KEY"

The following request captures the execution, forks it before the chosen action, and records the intended behavior:

curl -X POST https://api.runtype.com/v1/eval/suites/YOUR_SUITE_ID/cases/from-execution \
  -H "Authorization: Bearer YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "executionId": "YOUR_EXECUTION_ID",
    "forkMessageIndex": 4,
    "expected": { "text": "Look up the order before issuing a refund." }
  }'

Replace the following:

• YOUR_API_KEY: your Runtype API key
• YOUR_EXECUTION_ID: the ID of the Agent execution to capture
• YOUR_SUITE_ID: the eval suite that receives the case
• 4: a forkMessageIndex value from the preview. If you omit it, the fork is immediately before the last tool call. For a run with no tool calls, the case grades the agent’s reply.

The request also accepts optional name and notes fields. The response’s replayable field is false when some tool outputs were truncated at capture.

A suite with 50 or fewer enabled cases runs inline and returns its scores. A larger suite returns 202 with a runId. Read its scores from GET /v1/eval/runs/YOUR_RUN_ID/scores after it completes. To grade a saved version before it goes live, send {"agent": {"versionId": "YOUR_VERSION_ID"}} or {"agent": {"alias": "YOUR_ALIAS"}} in the run request body.

The TypeScript SDK exposes the same calls as Runtype.evals.suites.getCapturePreview, addCaseFromExecution, and run. For request and response shapes, see the API reference.

Troubleshooting

The following problems are the most common:

• 403: The API key lacks a permission. Reading Agent logs and runs needs AGENTS:READ. Capture needs EVALS:WRITE. The capture preview needs EVALS:READ, EVALS:WRITE, or AGENTS:READ. Running a suite needs FLOWS:EXECUTE or EVALS:WRITE.
• 404 with “not found or not accessible” on capture or preview: The execution is not an Agent execution, or it belongs to another account or organization.
• No failures in the list: The logs and runs endpoints return the last 24 hours unless you pass from. Runs older than your plan’s log retention period are not returned. For details, see Log retention.
• 402 on a suite run: The run exceeded a limit. With code: "DAILY_EVAL_LIMIT", retry after the daily eval limit resets at midnight UTC. With code: "SPEND_LIMIT_EXCEEDED", the organization reached its spend limit.

Agents that run outside Runtype

An agent built with Flue, the Vercel AI SDK, LangGraph, the OpenAI Agents SDK, Mastra, OpenClaw, Hermes, Pi, or a custom agent loop joins the improvement loop when it exports OpenTelemetry traces to Runtype. To set up the export, see Bring your own agent. For Flue on Cloudflare Workers, the export is a dashboard destination plus one wrangler.jsonc block, with no code.

For the Cloudflare Agents SDK, read its status first. The native path needs wrapAISDK, with storeMessages and storeTools set so that the run has a transcript to capture. Cloudflare can group several turns into one trace or split one request across several traces, and Runtype creates one run per trace.

The improvement loop differs in three places:

• Observe. Ingested runs show an External badge in the Runs view and carry otel-... execution IDs. Their timings and cost are what the agent reported, not what Runtype measured or billed.
• Capture. Capture needs the run’s transcript. The exported spans must include gen_ai.input.messages and gen_ai.output.messages. Cloudflare Workers traces, @flue/opentelemetry, and @runtypelabs/flue-otel 0.5 or later include them by default. If you set content: false on @runtypelabs/flue-otel, the spans carry no transcript and you can’t capture the run. The OpenClaw and Hermes plugins and the Pi coding agent extension send content only when you set RUNTYPE_CAPTURE_CONTENT=true (OpenClaw and Pi) or RUNTYPE_OTEL_CAPTURE_CONTENT=true (Hermes). The Pi durable binding sends content unless you pass content: false.
• Fix and verify. You fix the agent in your repository, not with update_agent. run_eval_suite works only when Runtype can call the agent. An external agent registered with an Agent2Agent (A2A) endpoint qualifies, but the suite skips cases that replay recorded tool results. For an agent that only sends telemetry or uses a runtype-stream endpoint, re-run the scenario against your redeployed agent. Then inspect the new run with trace_execution. The captured cases remain the regression record.

Give your coding agent a prompt like the following for an agent that runs outside Runtype:

My agent "AGENT_NAME" runs outside Runtype and sends its traces to Runtype.
It has returned wrong answers. Use the Runtype MCP tools to run an
improvement loop:
1. Find the failure. Call `list_runs` with `agent_id` set to the agent's ID
   and `status` set to `"failed"`. Pass `from` if the failure is older than
   24 hours.
2. Diagnose it. Call `trace_execution` and inspect the tree. Identify the
   iteration, tool call, or model turn that failed. Quote trace evidence before
   you draw a conclusion.
3. Capture it. Call `get_eval_capture_preview` to review fork points. Call
   `add_eval_case_from_execution` to save the execution in the agent's eval
   suite. If no suite exists, call `create_eval_suite` first. Write the
   expected behavior, not the observed incorrect output.
4. Fix it. Propose the change to my agent's code and show me the diff.
5. Verify it. Tell me which scenario to re-run, then inspect the new run
   with `trace_execution`.
After each run, report what you found and what you changed.

Replace AGENT_NAME with the name of the Runtype Agent that owns the ingested runs.

Next steps

Continue the improvement loop with these guides:

• Agent versioning and publishing: grade a version before it goes live
• Running an eval: choose an execution mode and review results
• Interpreting eval results: read pass and fail outcomes and grader evidence
• Managing eval suites: manage cases, graders, runs, and captured-case replay
• Designing evals that work: identify failure modes and write graders
• Working with Logs: inspect execution details, traces, and filters
• Manage evals as code: run suites from your repository and CI
• Reporting telemetry from an external agent: send OpenTelemetry traces from your own infrastructure
• Runtype MCP server: review the tools and connection options
• Braintrust (tracing): send execution spans to your observability workspace
• Querying logs programmatically: query logs with the API, runtype logs, and runtype tail
• Working with runs: find runs and read the run detail page