> For clean Markdown of any page, append .md to the page URL. > For a complete documentation index, see https://docs.runtype.com/llms.txt. > For AI client integration (Claude Code, Cursor, etc.), connect to the MCP server at https://docs.runtype.com/_mcp/server. # Designing evals that work Design eval suites around observed failures. Make each case test a behavior that matters to you, and define what each grader checks. Use Evals to track whether your changes fix those failures without introducing new ones. ## Start with error analysis, not a checklist Before you write cases, inspect what your Flow or Agent does on realistic inputs. Don't start with a list of hypothetical failures. Start with observed runs. Use this process: 1. Run your Flow or Agent on realistic inputs. 2. In [Logs](/user-guide/logs/what-are-logs), read the outputs and traces of those runs and of recent production runs. Look for recurring patterns, not only scores. 3. Name and count the failure modes. Write down examples such as "invents a refund amount", "answers in the wrong language", or "ignores the account tier". Use this list to choose cases and graders. If a grader doesn't map to an observed failure or a concrete risk, you can't tell which decision its score supports. ## Use binary pass or fail checks Define each case against a specific pass or fail question. Each grader has a severity. A gate grader miss fails the case. A soft grader miss is reported but doesn't fail the case. To make a grader soft, turn on **Warn instead of fail** when you edit it. To count soft misses as failures, open the run drop-down menu on the suite page and click **Run in strict mode**, or pass `--strict` to `runtype eval run`. The suite score is the number of passed cases divided by the number of graded cases. Cases that the platform stopped before your Flow or Agent answered aren't counted. For details, see [Managing eval suites](/user-guide/evals/managing-eval-suites). Keep one obligation in each grader. If a criterion contains the word "and", split the obligations. For example, "answers the question and uses a professional tone" contains two checks. On an AI grader, **Split into focused checks** proposes one sub-check for each obligation. Review the proposals before you accept them as separate graders. ## Choose the least expensive grader that fits An AI grader adds a model call for each case and can disagree with your judgment. Start with deterministic checks, then run-behavior checks, and then AI graders. Use this order: * **Deterministic output checks**: choose **Output contains text**, **Matches the expected answer**, or **Valid JSON** when a known string, schema, or format defines the correct output. * **Run-behavior checks**: choose **Called a tool**, **Completed without errors**, or **Cost under a limit** when you need to assert what the run did. * **AI graders**: use these for qualities that deterministic and run-behavior checks can't express, such as tone, relevance, or whether the answer addresses the question. Choose a deterministic check when you can state a precise condition. For example, use an **Output contains text** check when an output fails whenever it omits a ticket number. Use an AI grader when the requirement allows valid rewording, such as two answers that convey the same meaning. To accept reworded answers, turn on **Compare against the expected answer** so that the AI grader uses the case's expected answer as a reference. The **Add a grader** dialog offers ready-made AI graders that you can start from and then edit. Several of them match common failure modes. For example, **Same language** catches answers in the wrong language, and **Grounded, no made-up facts** catches invented details such as a refund amount. Other presets include **Answers the question**, **Matches the expected answer**, and **Refuses when it should**. ## Validate an AI grader before you trust it An AI grader can disagree with your judgment. Calibrate it before you use it to gate a decision. Label 20 to 30 cases by hand as pass or fail. Run the AI grader on the same cases and compare its verdicts with your labels. If the grader disagrees often, rewrite its criteria and repeat the comparison. In a suite that you manage as code, you can also pin the grader's model. For details, see [AI judge grading](/user-guide/evals/manage-evals-as-code#ai-judge-grading). On a suite run's results page, use the thumbs-up (**Grader got this right**) and thumbs-down (**Grader got this wrong**) buttons on each AI-grader outcome. They record whether the verdict matches your judgment. The suite page shows **Judge trust** as a fraction such as "Agrees with you 7 of 9 times". Read the grader's reasoning and the evidence that it cites in the trace. **Insufficient evidence** means that the grader doesn't have enough information to decide. The results page displays this outcome as a warning, and it doesn't fail the case. A change in the grader's model can change the reported score without changing the quality of the system under test. Compare grader agreement before you trust a new score. To test the grader's stability, rerun the suite on slightly changed inputs. If the verdict changes, review the criteria. ## Capture production failures as regression cases Capture an intended-behavior failure before you fix it. Use this loop: 1. Reproduce the failure. 2. Inspect the run's trace. For a Flow, use the **Test** view of the **Run Flow** sheet. For an Agent, use the run inspector or Logs. 3. Add the failing input as a case. Add graders or an expected answer that define the correct behavior. 4. Run the current version and confirm that the case fails. 5. Fix the Flow or Agent. 6. Run the suite again. Confirm that the case passes and that existing cases still pass. For a Flow, click **Save as test case** after a test run in the **Run Flow** sheet. The first saved case creates an eval suite for the Flow. For more information, see [Debugging flows](/user-guide/flows/debugging-flows). For an Agent run, click **Add to evals** in the run inspector, or **Add to eval** in Logs. In the **Eval** list, choose a suite for the Agent. If no suite targets the Agent, Runtype creates a default suite when you save. Under **Fork point**, choose the recorded action to grade. The conversation is frozen immediately before that action. If the dialog reports that the case is partially replayable, some tool outputs were too large to capture in full, and the case replays the truncated outputs. Assert intended behavior, not the current behavior. Don't paste a wrong current output into the expected answer. Exclude infrastructure failures such as rate limits and provider outages from regression cases. Remove personal or sensitive data from captured inputs before you save them. ## Prefer small suites of real failures Start with 5 to 15 realistic cases from real usage and edge cases that you care about. Add cases from production failures over time. Prioritize real failures over synthetic cases that cluster around routine inputs. Proposed cases go to the suite's **Review queue**, and no proposal is added until you accept it. Prefer proposals marked **From a run** or **From feedback**. Use **Generate cases** sparingly, and review each proposal before you accept it. To find tools and instructions that no case tests, open the suite's **Coverage** panel. For details, see [Managing eval suites](/user-guide/evals/managing-eval-suites#view-a-suite). Apply these rules when you assess suite size and results: * Treat a suite with eight cases and a change from 6/8 to 7/8 as limited evidence. Report the suite size with the result. * Treat a passing suite as a regression harness, not proof of production readiness. * When a suite scores 100%, add cases for known difficult scenarios instead of treating the score as complete coverage. ## Test each step and the final result Check each step of a multi-step Flow as well as its final result. A final result doesn't show where an earlier failure began. If eight steps each succeed 95% of the time, all eight succeed about 66% of the time when their outcomes are independent. Add a **Ran a step** grader to assert that a specific step ran, and a **Step order** grader to assert that steps ran in the required order. For tool calls, add a **Tool order** grader. Keep at least one end-to-end case that checks the complete result. When a step changes state, verify the state change itself. A message that claims a Record was saved doesn't prove that the Record was saved. Graders check the run and its output, not stored Records. To verify a saved Record, make the Flow output the saved Record and check it with a **JSON field** grader. The run results page shows grader outcomes beside the trace, and evidence chips identify the step or tool call that an AI grader cites. Use this view to identify the stage that failed. If you rely on repeated behavior, run the suite multiple times and require each run to pass. Don't use an average score as your only requirement. ## Decide when to build a suite Build a suite when you need to check a behavior repeatedly. If you can fix a failure immediately, fix it first. Then capture a regression case so that you can check the fix again. ## Keep your judgment in the loop The platform surfaces traces, proposes cases, splits multi-part criteria, and runs comparisons. You define the failure labels, the criteria, and the tie-breaking rules for ambiguous cases. Review proposed cases and grader results before you add a case or use a grader as a gate. ## Next steps Continue with one of these guides: * [What are Evals?](/user-guide/evals/what-are-evals): learn the concepts behind suites, cases, and graders. * [Running an eval](/user-guide/evals/running-an-eval): compare Flow configurations against test data. * [Managing eval suites](/user-guide/evals/managing-eval-suites): configure graders, review cases, and read run results. * [Interpreting eval results](/user-guide/evals/interpreting-eval-results): read a run and decide what to change. * [Maintaining your eval set](/user-guide/evals/maintaining-your-eval-set): choose the metrics worth keeping and retire the rest. * [Manage evals as code](/user-guide/evals/manage-evals-as-code): define suites in your repository and gate CI on them.