> For clean Markdown of any page, append .md to the page URL.
> For a complete documentation index, see https://docs.runtype.com/llms.txt.
> For AI client integration (Claude Code, Cursor, etc.), connect to the MCP server at https://docs.runtype.com/_mcp/server.

# Maintaining your eval set

Treat each grader as a metric that you run, review, and use to make a decision. Suites lose value when metrics accumulate without a clear purpose and their scores stop guiding changes. Choose metrics that deserve a place in your suite, then keep the set small as your product changes. For information about writing graders, see [Designing evals that work](/user-guide/evals/designing-evals-that-work).

## Three roles a metric can play

Each metric in your suite plays one of three roles, as described in the following table:

| Role               | Question it answers                                | Typical source                           |
| ------------------ | -------------------------------------------------- | ---------------------------------------- |
| Goal metric        | Is quality improving toward your product goal?     | Error analysis, product goals            |
| Guardrail          | Did a change break a requirement that must hold?   | Requirements, compliance, past incidents |
| Operational metric | What does the run cost, and how long does it take? | Recorded on every run                    |

Assess your suite against all three roles. Push goal metrics toward your product goal, use guardrails to protect requirements, and use operational metrics to explain the other two. In Runtype, these roles map to grader severity and the run record:

* **Guardrails: gate graders.** A gate miss fails its case. When you [run a suite as a CI gate](/user-guide/evals/manage-evals-as-code#run-as-a-ci-gate), a failing suite blocks the merge. Add gate graders for compliance rules, safety requirements, and format contracts from the start. For example, write "The agent never calls `delete_record`" as a `notCalledTool('delete_record')` grader in code, or add **Did not call a tool** from **Check the run** in the dashboard.
* **Goal metrics: AI graders, often soft at first.** Use [AI graders](/user-guide/evals/manage-evals-as-code#ai-judge-grading) to check faithfulness to retrieved context and whether the reply resolves the request. Mark an aspirational grader soft: use `.soft()` in code, or turn on **Warn instead of fail** in the dashboard. A soft miss is reported but doesn't fail its case unless you run in strict mode. Promote the grader to a gate after it reaches the level you want to defend.
* **Operational metrics: recorded run data, plus optional soft budgets.** Every run records latency and cost without a grader. When a budget carries a real decision, add `latency(3000).soft()` or `cost(0.05).soft()` in code, or **Responded in time** or **Cost under a limit** with **Warn instead of fail** in the dashboard. Review the remaining run data without adding graders.

Keep only the metrics that inform a decision. Each grader adds another result to run and review.

### Track goal metrics over time

The score-over-time chart in the suite's **Runs** section plots the suite score: passed cases out of total cases. It appears when the current page has at least two scored runs. On a normal run, a soft miss doesn't fail its case, so soft graders don't move the chart. Track a soft goal metric in one of these ways:

* Open a run's results page and read the **N soft misses** badge on each passed case.
* Turn on **Strict grading** on the run's results page to count soft misses in the headline score.
* Run the suite in strict mode, where soft misses fail their cases. In the dashboard, open the **Run options** menu next to **Run** and click **Run in strict mode**. In CI, run `runtype eval run --strict`.

To keep a reference point, open a completed run's row menu in the **Runs** list and click **Pin as baseline**. The pinned run shows a **Baseline** badge. Compare later runs against it yourself. Runtype doesn't compare runs against the baseline automatically. For more information, see [View a suite](/user-guide/evals/managing-eval-suites#view-a-suite).

## Where candidate metrics come from

Look for candidates in the following order:

1. **Start with observed failures.** Read traces, name what went wrong, and use the error-analysis approach in [Designing evals that work](/user-guide/evals/designing-evals-that-work#start-with-error-analysis-not-a-checklist). Write graders for failures you observed, not failures you imagined. The [review queue](/user-guide/evals/managing-eval-suites#view-a-suite) collects proposed cases from generation, coverage gaps, individual runs, and conversation feedback.
2. **Add goals and hard constraints.** Define a small set of requirements before the first trace exists. Compliance rules, safety requirements, and format contracts become guardrail graders from the start.

Use grader catalogs to explore options, not to choose final criteria. Generic qualities such as helpfulness or toxicity measure abstract properties that don't match every product's failure modes. Start with a ready-made AI grader, such as **Grounded, no made-up facts** in the dashboard or [`judges.grounded()`](/user-guide/evals/manage-evals-as-code#graders) in code. Then rewrite its criteria in the language of your product and its observed failures.

## Which candidates deserve a grader

Not every named failure mode deserves a permanent metric. Apply three filters before you add a grader.

### Separate one-time fixes from recurring failures

Some failures result from missing instructions. If a prompt change resolves a failure, make the change. [Capture the failing input as a regression case](/user-guide/evals/designing-evals-that-work#capture-production-failures-as-regression-cases) with a deterministic check. Skip a separate recurring metric. Examples include invalid JSON output, an incorrect date format, and markdown on a plain-text channel.

Keep a metric for a failure that a prompt change can't resolve and that you need to track across many inputs. Examples include support from retrieved context, correct context retrieval, and request resolution. Also check whether the agent selected the right tool with the right arguments.

### Tie every metric to a decision

For each candidate, name the action that changes when the number moves. Examples include blocking a deploy, rolling back a prompt, and opening an investigation. If no action changes, the metric is noise and doesn't belong in the suite.

A gate grader meets this filter by definition: a gate miss fails the case, and in a CI gate a failing suite blocks the merge. Assign each soft grader an owner and a threshold that triggers an action. Don't add numbers without a decision behind them. Conversation length can rise when users are engaged or stuck, so you can't act on it alone.

### Manage judge costs

Prefer deterministic output checks and [trace graders](/user-guide/evals/manage-evals-as-code#trace-graders) when they capture the failure. They don't add a model call. Each AI grader adds one model call per case per run, plus criteria to maintain and [judge-trust reviews](/user-guide/evals/designing-evals-that-work#validate-an-ai-grader-before-you-trust-it) to keep current. Cut an expensive metric when it doesn't support a strong decision. Use the [grader ladder](/user-guide/evals/designing-evals-that-work#choose-the-least-expensive-grader-that-fits) to find a cheaper check that captures the same failure.

## Maintain the set

Your product changes, so update the metric set when its behavior changes. The following habits keep the set useful:

* **Re-run error analysis after behavior changes.** Prompt rewrites, model changes, and new tools change the failure distribution. After you change the target's definition, open the suite's **Coverage** section and click **Check coverage** (or **Refresh**). The section lists the target's current tools and instructions and marks which ones your cases exercise. To propose cases for an uncovered item, click **Generate** next to it. Generated cases wait in the review queue until you accept them. Each generation counts once against your daily eval limit, and each suite accepts at most 50 new proposals in a rolling 24-hour period.
* **Retire metrics that stop catching failures.** Runtype has no per-grader pass-rate view across runs. To find graders that pass on every run, open recent runs and read each grader's outcome, or call `Runtype.evals.getRunScores(runId)` for recent runs and count misses per grader. For a non-guardrail grader, remove it or combine its case with a broader one when it stops finding failures. A guardrail can remain at 100% when it protects a constraint that must hold. For a suite that you manage in the dashboard, click the **Remove** button on the grader row. For a suite managed in code, delete the grader from your `*.eval.ts` file. The next [`ensure` call in your deployment](/user-guide/evals/manage-evals-as-code#converge-at-deploy-time) updates the suite to match your code.
* **Review the metrics that you optimize.** When you tune prompts against a judge, you can optimize for the judge instead of your users. On a run's results page, use the thumbs-up and thumbs-down buttons next to **Grader got this** on each AI-grader outcome to record whether the grader was right. In code, call `Runtype.evals.reviewScore(scoreId, verdict)`. The suite's **Judge trust** value summarizes these reviews across all AI graders in the suite. If it falls, find the disagreements on recent run pages to see which grader's criteria to rewrite, and rewrite them before you rely on more AI-grader scores.

> **Warning**
>
> When a code-managed suite's definition changes, the next `ensure` call replaces all of the suite's cases with the cases in your repository. This removes cases that you accepted from the review queue or added in the dashboard. Before you change the definition in code, copy those cases into your `*.eval.ts` file. To fetch the suite's current definition, including its cases, call `Runtype.evals.pull(name)`. For more information, see [Handle dashboard edits](/user-guide/evals/manage-evals-as-code#handle-dashboard-edits).

## Next steps

Continue with these guides:

* [Designing evals that work](/user-guide/evals/designing-evals-that-work): error analysis, the grader ladder, and validating an AI judge
* [Managing eval suites](/user-guide/evals/managing-eval-suites): the suite page, review queue, coverage, and judge trust
* [Manage evals as code](/user-guide/evals/manage-evals-as-code): gate and soft severity, and running suites as a CI gate
* [Improving your agents from production traces](/developer-guides/guides/improving-agents-from-production): the trace-reading loop that feeds error analysis
* [Interpreting eval results](/user-guide/evals/interpreting-eval-results): read a run and decide what to change