Maintaining your eval set

Treat each grader as a metric that you run, review, and use to make a decision. Suites lose value when metrics accumulate without a clear purpose and their scores stop guiding changes. Choose metrics that deserve a place in your suite, then keep the set small as your product changes. For information about writing graders, see Designing evals that work.

Three roles a metric can play

Each metric in your suite plays one of three roles, as described in the following table:

Assess your suite against all three roles. Push goal metrics toward your product goal, use guardrails to protect requirements, and use operational metrics to explain the other two. In Runtype, these roles map to grader severity and the run record:

• Guardrails: gate graders. A gate miss fails its case. When you run a suite as a CI gate, a failing suite blocks the merge. Add gate graders for compliance rules, safety requirements, and format contracts from the start. For example, write “The agent never calls delete_record” as a notCalledTool('delete_record') grader in code, or add Did not call a tool from Check the run in the dashboard.
• Goal metrics: AI graders, often soft at first. Use AI graders to check faithfulness to retrieved context and whether the reply resolves the request. Mark an aspirational grader soft: use .soft() in code, or turn on Warn instead of fail in the dashboard. A soft miss is reported but doesn’t fail its case unless you run in strict mode. Promote the grader to a gate after it reaches the level you want to defend.
• Operational metrics: recorded run data, plus optional soft budgets. Every run records latency and cost without a grader. When a budget carries a real decision, add latency(3000).soft() or cost(0.05).soft() in code, or Responded in time or Cost under a limit with Warn instead of fail in the dashboard. Review the remaining run data without adding graders.

Keep only the metrics that inform a decision. Each grader adds another result to run and review.

Track goal metrics over time

The score-over-time chart in the suite’s Runs section plots the suite score: passed cases out of total cases. It appears when the current page has at least two scored runs. On a normal run, a soft miss doesn’t fail its case, so soft graders don’t move the chart. Track a soft goal metric in one of these ways:

• Open a run’s results page and read the N soft misses badge on each passed case.
• Turn on Strict grading on the run’s results page to count soft misses in the headline score.
• Run the suite in strict mode, where soft misses fail their cases. In the dashboard, open the Run options menu next to Run and click Run in strict mode. In CI, run runtype eval run --strict.

To keep a reference point, open a completed run’s row menu in the Runs list and click Pin as baseline. The pinned run shows a Baseline badge. Compare later runs against it yourself. Runtype doesn’t compare runs against the baseline automatically. For more information, see View a suite.

Where candidate metrics come from

Look for candidates in the following order:

1. Start with observed failures. Read traces, name what went wrong, and use the error-analysis approach in Designing evals that work. Write graders for failures you observed, not failures you imagined. The review queue collects proposed cases from generation, coverage gaps, individual runs, and conversation feedback.
2. Add goals and hard constraints. Define a small set of requirements before the first trace exists. Compliance rules, safety requirements, and format contracts become guardrail graders from the start.

Use grader catalogs to explore options, not to choose final criteria. Generic qualities such as helpfulness or toxicity measure abstract properties that don’t match every product’s failure modes. Start with a ready-made AI grader, such as Grounded, no made-up facts in the dashboard or judges.grounded() in code. Then rewrite its criteria in the language of your product and its observed failures.

Which candidates deserve a grader

Not every named failure mode deserves a permanent metric. Apply three filters before you add a grader.

Separate one-time fixes from recurring failures

Some failures result from missing instructions. If a prompt change resolves a failure, make the change. Capture the failing input as a regression case with a deterministic check. Skip a separate recurring metric. Examples include invalid JSON output, an incorrect date format, and markdown on a plain-text channel.

Keep a metric for a failure that a prompt change can’t resolve and that you need to track across many inputs. Examples include support from retrieved context, correct context retrieval, and request resolution. Also check whether the agent selected the right tool with the right arguments.

Tie every metric to a decision

For each candidate, name the action that changes when the number moves. Examples include blocking a deploy, rolling back a prompt, and opening an investigation. If no action changes, the metric is noise and doesn’t belong in the suite.

A gate grader meets this filter by definition: a gate miss fails the case, and in a CI gate a failing suite blocks the merge. Assign each soft grader an owner and a threshold that triggers an action. Don’t add numbers without a decision behind them. Conversation length can rise when users are engaged or stuck, so you can’t act on it alone.

Manage judge costs

Prefer deterministic output checks and trace graders when they capture the failure. They don’t add a model call. Each AI grader adds one model call per case per run, plus criteria to maintain and judge-trust reviews to keep current. Cut an expensive metric when it doesn’t support a strong decision. Use the grader ladder to find a cheaper check that captures the same failure.

Maintain the set

Your product changes, so update the metric set when its behavior changes. The following habits keep the set useful:

• Re-run error analysis after behavior changes. Prompt rewrites, model changes, and new tools change the failure distribution. After you change the target’s definition, open the suite’s Coverage section and click Check coverage (or Refresh). The section lists the target’s current tools and instructions and marks which ones your cases exercise. To propose cases for an uncovered item, click Generate next to it. Generated cases wait in the review queue until you accept them. Each generation counts once against your daily eval limit, and each suite accepts at most 50 new proposals in a rolling 24-hour period.
• Retire metrics that stop catching failures. Runtype has no per-grader pass-rate view across runs. To find graders that pass on every run, open recent runs and read each grader’s outcome, or call Runtype.evals.getRunScores(runId) for recent runs and count misses per grader. For a non-guardrail grader, remove it or combine its case with a broader one when it stops finding failures. A guardrail can remain at 100% when it protects a constraint that must hold. For a suite that you manage in the dashboard, click the Remove button on the grader row. For a suite managed in code, delete the grader from your *.eval.ts file. The next ensure call in your deployment updates the suite to match your code.
• Review the metrics that you optimize. When you tune prompts against a judge, you can optimize for the judge instead of your users. On a run’s results page, use the thumbs-up and thumbs-down buttons next to Grader got this on each AI-grader outcome to record whether the grader was right. In code, call Runtype.evals.reviewScore(scoreId, verdict). The suite’s Judge trust value summarizes these reviews across all AI graders in the suite. If it falls, find the disagreements on recent run pages to see which grader’s criteria to rewrite, and rewrite them before you rely on more AI-grader scores.

Next steps

Continue with these guides:

• Designing evals that work: error analysis, the grader ladder, and validating an AI judge
• Managing eval suites: the suite page, review queue, coverage, and judge trust
• Manage evals as code: gate and soft severity, and running suites as a CI gate
• Improving your agents from production traces: the trace-reading loop that feeds error analysis
• Interpreting eval results: read a run and decide what to change