Improving your agents from production traces

Use production traces to improve an agent. Read a real execution, capture a failure as an eval case, change the agent, and run the suite to check the result.

Runtype records Flow and Agent executions in Logs. A trace includes agent iterations, model calls, tool invocations, flow steps, duration, token usage, and cost. You do not add instrumentation to inspect executions that Runtype runs. Agents that run elsewhere join the same loop by sending their traces to Runtype; see Agents that run outside Runtype. Evals support deterministic checks, agent trace checks, and AI graders, so you can turn a production failure into a regression case.

This guide uses the following loop:

1. Observe. Find an execution that failed or returned a wrong answer.
2. Diagnose. Read the trace and locate the iteration, tool call, or model turn that failed.
3. Capture. Save the execution as an eval case before changing the agent.
4. Fix. Change the agent’s instructions, tools, or model.
5. Verify. Run the suite. Confirm that captured and existing cases pass.

Use the dashboard or connect a coding agent to the Runtype Model Context Protocol (MCP) server.

Why start with production failures

Use these practices when you build or maintain an eval suite:

• Start with real failures. Use production outputs to identify failure modes instead of inventing cases from a brainstorm. Name and count each mode.
• Capture before changing the agent. A captured case records the failure before a prompt, tool, or model change. Use the case to check whether the change fixes the failure.
• Record intended behavior. Write the correct behavior in the expected output or grader criteria. Do not copy the incorrect output. Exclude provider outages and other service failures from regression cases.

Use the dashboard

To find a candidate execution, open Logs. Use the Status, Type, and Time range controls to narrow the list. Presets range from 15 minutes to 30 days. Choose Custom range for a specific interval. You can also select a range in the timeline chart or turn on Live. To inspect a failed execution, select a row with Failed status.

Use the Details tab to inspect each step’s input and output. Use the Trace tab to inspect the execution tree. The trace includes iterations, tool calls, and model turns. Use these signals:

• Failed nodes: Find the node with an error status and inspect its input and error message.
• Tool calls: Compare tool names and arguments with the intended behavior.
• Context per iteration: Review estimated input tokens across iterations. When the model context size is known, the chart marks its ceiling.

To capture the execution, click Add to eval in the detail view. Select an eval, or let Runtype create a default eval for the agent. If the run includes tool calls, use Fork point to select the action that the eval grades next. If the run has no tool calls, the eval grades the agent’s reply. Click Add to eval.

The case freezes the conversation at the fork point and grades the agent’s next step. Recorded tool results are stored as editable mocks. If a tool output is too large, the case is partially replayable.

To fix the failure, change the agent’s instructions, tools, or model. On the eval suite page, click Run. Review the run results to confirm that the captured case passes and existing cases do not regress. When an AI grader cites evidence, inspect its reasoning and the cited evidence. Use the thumbs-up or thumbs-down control to record whether the verdict matches your judgment. The Judge trust stat shows the number of agreement reviews and total reviews.

Use the MCP server

Use the Runtype MCP server to run this loop from Claude Code, Cursor, or another MCP client. Call list_logs or list_agent_executions to find candidate executions. Call get_log_stats to review aggregate log counts. Call trace_execution or trace_conversation to inspect traces. Call get_eval_capture_preview before add_eval_case_from_execution to choose a fork point. Call update_agent to apply a change and run_eval_suite to verify it.

To configure Claude Code manually, run:

$
claude mcp add --transport http runtype https://api.runtype.com/v1/mcp/protocol

For an all-in-one setup, run npx -y @runtypelabs/cli@latest onboard. It configures the MCP connection and installs Runtype agent skills. For details, see Connect Runtype to coding agents.

Give your coding agent a prompt like the following:

My Runtype agent "AGENT_NAME" has returned wrong answers. Use the
Runtype MCP tools to run an improvement loop:
1. Find the failure. Call `list_logs` with `level` set to `"error"` and
   `order` set to `"desc"`. Use `list_agent_executions` to find the agent's
   most recent failed execution or wrong answer.
2. Diagnose it. Call `trace_execution` and inspect the tree. Identify the
   iteration, tool call, or model turn that failed. Quote trace evidence before
   you draw a conclusion.
3. Capture it. Call `get_eval_capture_preview` to review fork points. Call
   `add_eval_case_from_execution` to save the execution in the agent's eval
   suite. If no suite exists, call `create_eval_suite` first. Write the
   expected behavior, not the observed incorrect output.
4. Fix it. Propose a change to the agent's instructions or tool configuration.
   Show me the change. Apply it with `update_agent` after I approve it.
5. Verify it. Call `run_eval_suite`. Confirm that the captured case passes and
   existing cases do not regress. If a case fails, revise the change and run
   the suite again.
After each run, report what you found, what you changed, and the suite score
before and after.

Replace AGENT_NAME with your Runtype Agent name.

Repeat this sequence after each production incident. Each captured case adds a concrete failure to the suite and gives later changes a case to check.

Use the REST API

Use these REST endpoints to automate the loop from your own tooling:

Replace AGENT_ID, EXECUTION_ID, CONVERSATION_ID, and SUITE_ID with IDs from your account. Send a bearer token with each request. A trace response contains a hierarchical tree of iteration, model, tool, and step nodes. It also returns aggregate totalDuration, totalTokens, and totalCost values. For request and response shapes, see the API reference.

Agents that run outside Runtype

An agent built with Flue, the Vercel AI SDK, LangGraph, the OpenAI Agents SDK, Mastra, or your own loop enters this loop by exporting OpenTelemetry traces to Runtype. Set up the export with Bring your own agent; for Flue on Cloudflare Workers it is a dashboard destination plus one wrangler.jsonc block, with no code. The Cloudflare Agents SDK needs wrapAISDK and records a whole WebSocket connection as one run; see its status first.

The loop then differs in three places:

• Observe. Ingested runs show an external badge in the Runs view and carry otel-... execution IDs. Their timings and cost are what the agent reported, not what Runtype measured or billed.
• Capture. add_eval_case_from_execution needs the run’s transcript, which requires conversation content on the exported spans (gen_ai.input.messages and gen_ai.output.messages). Workers traces, @flue/opentelemetry and @runtypelabs/flue-otel >= 0.5 all include it by default (content: false on @runtypelabs/flue-otel turns it off, and with it capture).
• Fix and verify. The fix is a change in your repository, not update_agent. run_eval_suite needs a target Runtype can execute: an external agent registered with an A2A endpoint qualifies (recorded-tool replay cases are skipped). For a telemetry-only or runtype-stream agent, re-run the scenario against your redeployed agent and inspect the resulting run with trace_execution; the captured cases remain the regression record.

Adjust the improvement-loop prompt from Use the MCP server accordingly: replace step 4 with “Propose the change to my agent’s code and show me the diff” and step 5 with “Tell me which scenario to re-run, then inspect the new run”.

Next steps

Continue the loop with these guides:

• Bring your own agent: connect an agent that runs on your own infrastructure
• Designing evals that work: identify failure modes and write graders
• Working with Logs: inspect execution details, traces, and filters
• Managing eval suites: manage cases, graders, runs, and captured-case replay
• Runtype MCP server: review the tools and connection options
• Braintrust (tracing): send execution spans to your observability workspace