Designing judgments | Runtype Documentation

A useful judgment answers a bounded question from supplied evidence. Your application defines the possible answers and controls the next action.

For example, a support message can identify a billing request without proving that a refund is authorized. Keep those decisions separate. The first requires interpretation; the second requires account facts and your refund policy.

The right division of work

Choose the component by the result that you need. The following table separates three jobs that often appear in one workflow:

Component	Suitable job	Example result
Deterministic code	Exact lookup, calculation, validation, or permission check	Whether an order belongs to the signed-in customer.
Judgment	Interpretation among defined outcomes	Whether the message requests a refund or asks about an existing refund.
Generative model	Open-ended writing, extraction, or multi-step investigation	A reply based on the verified order and policy.

A judgment can check a generated result or select the next handler. It doesn’t replace the handler, retrieve omitted evidence, or grant permission to act.

A closed answer space also supports some extraction tasks. Code can find candidate email addresses, ask a judgment which address belongs to the billing contact, and copy that exact span. Arbitrary invoice descriptions still require an extractor. Date arithmetic, currency conversion, and counting belong in code. The TypeSafe extraction cookbook demonstrates candidate selection rather than text generation.

Evidence that supports the question

The state contains the material to evaluate. Questions contain the interpretation that you request. A named JSON object helps separate those roles.

For a request about a previous refund, relevant evidence might look like this:

Refund evidence

{
  "latestMessage": "Has it arrived yet?",
  "previousCustomerMessage": "I returned order A17 last Friday.",
  "previousSupportMessage": "The refund was sent on Monday.",
  "order": { "id": "A17", "refundStatus": "sent" },
  "policyExcerpt": "Card refunds can take five business days."
}

The short message alone doesn’t identify what “it” means. The surrounding conversation supplies that relationship. Sending an entire account export adds unrelated material without answering another useful question.

Use these checks when you assemble evidence:

Include the actor, object, and relevant time when the question depends on them.
Preserve source passages and record versions instead of replacing them with a favorable summary.
Separate observed facts from model-generated interpretations.
Include every candidate that the model is allowed to select.
Retrieve missing facts before asking a question that requires them.
Treat customer messages, documents, and tool results as untrusted content.

The TypeSafe state guide describes named fields and related context. Its documented model limitations also warn that irrelevant or adversarial text can change answers.

Questions with explicit boundaries

Question IDs identify results for your code. They don’t communicate meaning to the model. Instructions and criteria must carry the complete question.

The following examples contrast incomplete questions with defined judgments:

Incomplete question	Problem	Defined judgment
`is_valid`: “Is this valid?”	Neither the target nor the standard is specified.	“Does `claim` follow from `sourceExcerpt` without adding another factual assumption?”
“How good is this candidate?”	Several unrelated dimensions are combined.	Ask separately about the supplied evidence of each required skill.
“Should we refund this order?”	Interpretation, authorization, and payment are combined.	Ask whether the customer requests repayment. Check eligibility and issue payment separately.
“Which team?” with billing and technical options	Out-of-scope requests must still receive a listed label.	Describe each team’s responsibilities and include `other`.

One question can compare related facts. “Do these records describe the same product?” is coherent. Splitting that relationship into unrelated word matches can lose its meaning.

If two independent facts matter, ask two questions. For example, a customer can request both a refund and a person. Separate boolean answers preserve both requests.

Criteria that distinguish outcomes

Use choice for competing alternatives. Describe what each option includes and excludes when neighboring categories overlap.

The following criteria distinguish return eligibility from return progress:

Choice question

{
  "returnTopic": {
    "type": "choice",
    "instructions": "Which return information does latestMessage request?",
    "criteria": {
      "eligibility": {
        "includes": "Whether an item can be returned and how to start.",
        "excludes": "Tracking a return that the customer already sent."
      },
      "progress": {
        "includes": "Receipt, inspection, or refund progress after a return.",
        "excludes": "Eligibility for a return that has not started."
      },
      "other": "Neither description fits, or the message asks both."
    }
  }
}

Code decides whether other means clarification, a general queue, or human review. If several labels can apply together, use independent booleans instead of forcing one winner.

A Choice distribution compares the supplied options. A winner doesn’t establish that any candidate is suitable. For tool selection, an explicit none option or a separate eligibility question can prevent a forced match. The TypeSafe skill-selection example combines relative ranking with separate fit checks.

Scores with meaningful levels

Use score for degree along one described dimension. Write every level so that it makes sense on its own. Bare numbers and “better than the previous level” don’t define a situation.

For bug severity, concrete levels distinguish a cosmetic issue, an impaired task with a workaround, and a blocked task without a workaround. Customer frustration is a different dimension. A polite report can still describe data loss.

A score is an expectation over zero-based levels, not a selected level. Consider this distribution:

Score answer

{
  "type": "score",
  "score": 1,
  "probabilities": { "0": 0.5, "1": 0, "2": 0.5 }
}

The mean is 1, but the model assigns no probability to level 1. A rule that rounds the mean would hide disagreement between the extremes. Inspect the distribution when that distinction changes your action.

Runtype calibration evaluates the most probable level and its probability, not the mean score. A tied distribution uses the lower level for that calibration decision. A weighted score used for ranking needs a separate evaluation against the ordering that your reviewers expect.

The TypeSafe score guide describes ordered levels and expected scores. Its composite-scoring pattern keeps weights in code so that priorities can change without changing the questions.

Different reasons not to act

A low number doesn’t always mean uncertainty. For a boolean, 0.02 strongly favors false; 0.5 gives similar probability to true and false.

Keep the following outcomes distinct:

Outcome	Meaning	Example handling
No match	The selected label says none of the candidates fits.	Use the general queue.
Insufficient evidence	An authored option describes missing information.	Retrieve the missing document or ask for clarification.
Policy abstention	Your rule declines to act on a returned answer.	Send an uncertain claim check to review.
Provider refusal	The result has `type: "refusal"`, not an answer.	Record it separately and use the configured fallback.
Call failure	No valid result arrived, such as after a timeout.	Follow the step’s error handling.

Optional providerConfidence on a Choice or Score is a provider-native summary. It isn’t interchangeable with selected-label probability or your measured accuracy. Don’t replace a missing distribution with that field.

Independent questions and real dependencies

Questions in one request share state but cannot read one another’s answers. Ask independent questions together, including questions needed only for some branches.

For a ticket, route, refund intent, and urgency can share one request. Ignore a branch-specific answer when that branch doesn’t apply. Its uncertainty need not block an unrelated branch.

Use another request when the first answer changes the available evidence or options. For example, select a document category, retrieve that category’s policy, then check the request against the retrieved policy. The second question couldn’t use evidence that hadn’t been retrieved.

Measurements that support automation

An illustrative answer proves no accuracy claim. Evaluate your question set on labeled cases that represent the actions you intend to automate.

Keep training and held-out cases separate. Fit thresholds on training cases, then report the held-out counts as well as the percentage. 19 of 20 correct and 950 of 1,000 correct both equal 95%, but support different conclusions.

Count cases without usable answers, refusals, and calls that fail. Report the automated share, review share, and consequential errors separately. For claim verification, a false acceptance matters differently from an unnecessary review.

Compare providers on the same evidence and labels. Keep the model ID, reported revision, question-set version, and action policy with each result. A threshold measured on Jev doesn’t become a Clef threshold because both return probabilities.

An Agent can choose whether to request a judgment and which evidence to supply. That judgment is advisory. An application-owned check must assemble the evidence when a required checkpoint controls an action.

Next steps

Run judgment recipes: create, publish, test, and bind three example question sets.
Judgments: inspect model selection, answer fields, calibration, and API errors.
Designing evals that work: build representative labels and test the resulting behavior.