Improving your agents from production traces

The fastest way to improve an agent is not another round of prompt brainstorming. It is reading what the agent actually did on real traffic, and turning every real failure into a test that can never silently break again.

Runtype traces every execution automatically: agent iterations, model calls, tool invocations, flow steps, durations, token counts, and cost. There is nothing to instrument and no telemetry pipeline to stand up. Evals with deterministic checks, run-behavior assertions, and AI judges are built into the same platform, so the distance from “I found a bad run” to “that failure is now a regression test” is one click or one tool call.

This guide is the practitioner’s loop for using that data:

1. Observe. Find a run that failed or produced a wrong answer.
2. Diagnose. Read its trace and locate the iteration, tool call, or model turn that actually went wrong.
3. Capture. Freeze that run as an eval regression case before you touch anything.
4. Fix. Change the agent’s instructions, tools, or model.
5. Verify. Re-run the suite. The new case flips to pass; the old cases prove nothing else broke.

Run it by hand in the dashboard, or hand the whole loop to a coding agent like Claude Code over MCP. Both are below.

Why this loop beats intuition

Three habits make the loop work, and they are worth stating before the mechanics:

• Start from real failures, not hypothetical ones. Cases invented in a brainstorm test your imagination; cases captured from production test your agent. Read actual outputs until failure modes emerge, then name and count them.
• Capture before you fix. The instinct after diagnosing a bug is to jump straight to the prompt editor. Capture first: a fix without a regression test is a fix that can quietly un-happen in the next model swap or prompt edit.
• Assert what should have happened. When you write the expected behavior for a captured case, describe the correct behavior, never the wrong-but-current output. And capture failures of intended behavior, not infrastructure blips; a provider outage is not a regression case.

Designing evals that work goes deeper on all three.

The loop in the dashboard

Observe. Open Logs in the sidebar. Filter by status and type, narrow the time range (presets from 15 minutes to 30 days, or drag a range on the timeline chart), or toggle Live to watch executions as they land. Failed runs carry an error status; click one to open the execution detail panel.

Diagnose. The detail panel has two tabs: Details, with each step’s full input and output, and Trace, a tree of the run’s iterations, tool calls, and model turns in order, so you can follow the agent loop end to end. Look for:

• The failed node. Errors sit on the exact iteration, tool call, or step that broke, with the input that caused it.
• Wrong tool choices. The tree shows every tool the agent invoked, in order, with arguments. A correct final answer built on a wrong lookup is luck, not quality.
• Context growth. The “Context per iteration” chart shows how context accumulates across the loop, and flags runs creeping toward the model’s context limit.

Capture. Click Add to eval on the execution. Pick the target suite (or let Runtype create a default one for the agent), choose which recorded tool call the eval should grade as the next step, and save. Runtype freezes the conversation up to that fork point and attaches every recorded tool result as an editable mock, so the case replays the real situation without re-executing side effects.

Fix and verify. Make your change to the agent, then click Run on the suite page. The captured case should fail before your fix and pass after it. On the run’s results page, every AI-grader verdict shows its cited evidence next to the trace, with 👍/👎 buttons to record whether the judge got it right; those reviews accumulate into the suite’s Judge trust stat, so you know how far to trust the automated verdicts.

Handing the loop to a coding agent

Every step above is exposed as MCP tools, so Claude Code, Cursor, or any MCP client can run the loop for you: list_logs and get_log_stats to find failures, trace_execution and trace_conversation to read them, list_agent_executions to scan an agent’s history, get_eval_capture_preview and add_eval_case_from_execution to capture regressions, update_agent to apply the fix, and run_eval_suite to verify it.

Connect once:

$
claude mcp add --transport http runtype https://api.runtype.com/v1/mcp/protocol

Or run npx -y @runtypelabs/cli@latest onboard to set up the connection plus the Runtype agent skills in one command (see Connect Runtype to coding agents).

Then hand your agent the loop. A prompt like this works well as a starting point; swap in your agent’s name:

My Runtype agent "support-triage" has been giving wrong answers. Using the
Runtype MCP tools, run an improvement loop:
1. Find the failure. Use list_logs (level "error", newest first) and
   list_agent_executions for the agent. Pick the most recent run that failed
   or produced a wrong answer.
2. Diagnose it. Pull the run with trace_execution and walk the tree: which
   iteration, tool call, or model turn went wrong, and why? Quote the
   evidence from the trace before drawing conclusions.
3. Capture it. Preview fork points with get_eval_capture_preview, then save
   the run as a regression case with add_eval_case_from_execution into the
   agent's eval suite (create one with create_eval_suite if none exists).
   Write the expected behavior as what SHOULD have happened, not what the
   run actually did.
4. Fix it. Propose a change to the agent's instructions or tool
   configuration, show it to me, and apply it with update_agent once I
   approve.
5. Verify. Re-run the suite with run_eval_suite. The new case must pass and
   no existing case may regress. If it fails, revise the fix and re-run.
Report each pass through the loop: what you found, what you changed, and
the suite score before and after.

This composes: run it after every incident and the suite grows one case per real failure, which is exactly how a suite becomes representative of your real traffic and hard to regress.

The same loop over REST

Scripting the loop into your own tooling instead? The same operations, in order:

All take a bearer token. The trace response is a hierarchical tree of iteration, model, tool, and step nodes with aggregate totalDuration, totalTokens, and totalCost; a just-finished execution is traceable immediately. Full request and response shapes are in the API reference.

Next steps

• Designing evals that work: build suites from real failures, not brainstorms
• Working with Logs: the full Logs and trace UI reference
• Managing eval suites: suites, graders, runs, and how captured cases replay
• Runtype MCP server: everything a connected coding agent can do
• Braintrust (tracing): stream execution spans into your own observability stack