Evals

Maintaining your eval set

Treat each grader as a metric that you run, review, and use to make a decision. Suites lose value when metrics accumulate without a clear purpose and their scores stop guiding changes. Choose metrics that deserve a place in your suite, then keep the set small as your product changes. For information about writing graders, see Designing evals that work.

Three roles a metric can play

Keep metrics in three roles. The following table names each role:

RoleQuestion it answersTypical source
Goal metricIs quality improving toward your product goal?Error analysis, product goals
GuardrailDid a change break a requirement that must hold?Requirements, compliance, past incidents
Operational metricWhat does the run cost, and how long does it take?Recorded on every run

Use all three roles in a suite. Push goal metrics toward your product goal, use guardrails to protect requirements, and use operational metrics to explain the other two. In Runtype, these roles map to grader severity and the run record:

  • Use gate graders for guardrails. A gate miss fails its case. When you run a suite as a CI gate, a failing suite blocks the merge. Add gate graders for compliance rules, safety requirements, and format contracts from the start. For example, write “The agent never calls delete_record” as a notCalledTool trace grader and keep it as a gate.
  • Use goal metrics to track the quality you are improving. Use AI graders to evaluate faithfulness to retrieved context. Use them to check whether the reply resolves the request. Track these metrics on the suite page’s score-over-time chart. Pin a baseline run for comparison. Mark an aspirational grader .soft() to see misses without blocking. Promote it to a gate after it reaches the level you want to defend.
  • Use operational metrics when a budget affects a decision. Every run records latency and cost without a grader. Add latency(3000).soft() or cost(0.05).soft() when a budget carries a real decision. Review the remaining run data without adding graders.

Keep only the metrics that inform a decision. Each grader adds another result to run and review.

Where candidate metrics come from

Find candidates in two places. Use this order:

  1. Start with observed failures. Read traces, name what went wrong, and use the error-analysis approach in Designing evals that work. Write graders for failures you observed, not failures you imagined. The review queue collects machine-proposed cases from production feedback and coverage gaps.
  2. Add goals and hard constraints. Define a small set of requirements before the first trace exists. Compliance rules, safety requirements, and format contracts become guardrail graders from the start.

Use grader catalogs to explore options, not to choose final criteria. Generic qualities such as helpfulness or toxicity measure abstract properties that do not match every product’s failure modes. Start with a built-in judge preset, then rewrite its criteria in the language of your product and its observed failures.

Which candidates deserve a grader

Not every named failure mode deserves a permanent metric. Apply three filters before you add a grader.

Separate one-time fixes from recurring failures

Some failures result from missing instructions. If a prompt change resolves a failure, make the change. Capture the failing input as a regression case with a deterministic check. Skip a separate recurring metric. Examples include invalid JSON output, an incorrect date format, and markdown on a plain-text channel.

Keep a metric for a failure that a prompt change cannot resolve and that you need to track across many inputs. Examples include support from retrieved context, correct context retrieval, and request resolution. Also check whether the agent selected the right tool with the right arguments.

Tie every metric to a decision

For each candidate, name the action that changes when the number moves. Examples include blocking a deploy, rolling back a prompt, and opening an investigation. If no action changes, the metric is noise and does not belong in the suite.

A gate grader meets this filter by definition. A gate miss fails the case, and a failing suite blocks the pipeline. Assign each soft grader an owner and a threshold that triggers an action. Do not add numbers without a decision behind them. Conversation length can rise when users are engaged or stuck, so you cannot act on it alone.

Manage judge costs

Prefer deterministic output checks and trace graders when they capture the failure. They do not add a model call. Each AI grader adds one model call per case per run, plus criteria to maintain and judge-trust reviews to keep current. Cut an expensive metric when it does not support a strong decision. Use the grader ladder to find a cheaper check that captures the same failure.

Maintain the set

Your product changes, so update the metric set when its behavior changes. Use these three habits:

  • Re-run error analysis after behavior changes. Prompt rewrites, model changes, and new tools change the failure distribution. After you change the target’s definition, open the suite’s Coverage panel. The instruction analysis refreshes automatically, and each uncovered obligation offers a targeted case proposal.
  • Retire metrics that stop catching failures. Review graders that pass on every run. For a non-guardrail grader, remove it or combine its case with a broader one when it stops finding failures. A guardrail can remain at 100% when it protects a constraint that must hold. For a dashboard-managed suite, click Remove on the grader row. For a suite managed in code, delete the grader from your *.eval.ts file. The next deployment updates the suite to match your code.
  • Review the metrics that you optimize. When you tune prompts against a judge, you can optimize for the judge instead of your users. Use the 👍/👎 controls on AI-grader outcomes to collect human labels. If the Judge trust fraction falls, rewrite the criteria before you rely on more AI-grader scores.

Next steps

Continue with these guides: