Judgments

A judgment returns typed answers that your program can use: boolean probabilities, selected labels, or rubric scores. Some providers also return a refusal instead of an answer.

Use a judgment for a bounded interpretation, such as a queue assignment or a claim check. Use a prompt step for prose, open-ended extraction, or work that needs tools. See Designing judgments for evidence and criteria, or Run judgment recipes for complete REST examples.

Judgments and prompt steps compose. Generate with a model, judge the result, then branch in the Flow. Or judge first to route cheaply, so that only the tickets that need a larger model reach one.

Runtype supports decision models through Vercel AI Gateway, OpenAI, and Cloudflare. The default is TypeSafe Jev through Vercel AI Gateway. To call TypeSafe’s own API instead, see Calling TypeSafe directly.

How a judgment differs from a prompt step

A judgment separates questions from the code that acts on their answers. Its interface differs from a prompt step:

• There is no prompt. You write a question set: named questions, each with a type, instructions, and criteria. Model settings such as temperature, a system prompt, and tools do not apply.
• The answer is data. Runtype normalizes provider responses into shared answer types and preserves supplied probabilities. Store those answers to apply different thresholds without another model call.
• Costs depend on the route. State, instructions, and criteria contribute input tokens. Providers can also charge for output. Read the catalog pricing and select the evidence that the questions require.
• A probability is not permission. What a probability of 0.9 means on your data is something that you measure with labeled examples. For how Runtype stores that measurement, see Calibrate a question set version.

Keep the five quantities apart

Conflating these quantities is how a wrong decision gets a number that looks trustworthy. Each one answers a different question:

• The boolean probability: the estimate that a boolean question’s statement is true.
• The selected label’s probability: the probability at probabilities[choice], when the provider supplies a distribution.
• A distribution statistic: a number that you compute from probabilities, such as the margin between the top two labels. It is absent when the answer carries no distribution.
• The rubric score: the expectation over a score question’s ordered levels, on the rubric’s own scale. A 2.4 on a rubric of five levels is not a probability.
• The policy outcome: the action that your thresholds turn an answer into. Plan for three outcomes from the start: act, abstain, and escalate.

Boolean answers carry a probability, Choice answers carry a selected label, and Score answers carry a mean. Compute distribution statistics only when probabilities are available. The policy outcome is yours. For why, see Thresholds stay with the caller.

Decision models

GET /v1/decision-models lists supported decision models, their limits, input kinds, pricing, and credential requirements. The equivalent MCP tool is list_decision_models with no arguments. Code Mode exposes runtype.listDecisionModels().

The catalog describes supported routes, not account readiness. A test call performs credential, funding, and policy checks. The following model IDs identify the native routes and the default gateway model:

Other Vercel decision models can appear in the catalog. Select an ID from the response instead of using a chat-model alias.

All Runtype judgment routes accept text, JSON objects, and JSON arrays. Top-level numbers, booleans, and null are not valid state. Native provider support for images or other media doesn’t make those inputs available through this interface. A URL inside JSON is text, not an attached image.

The OpenAI and Cloudflare native routes reject the strict zero-data-retention preference. They don’t silently weaken that preference or switch providers. Use a compatible route or resolve your retention requirements before changing the setting.

Cloudflare uses the ordinary Workers AI binding response. Per-call HTTP status, retry headers, and provider request IDs are not available on that route. The model gateway does not automatically retry an unknown binding failure; caller timeouts and their bounded retry policy still apply.

Question sets

A question set is the versioned rubric that your organization owns. You create it once, publish a version, and reference it by its questionSetId from every place that asks the same questions. One stored rubric keeps question definitions consistent across callers. Action rules and thresholds stay in caller code.

Dashboard availability

The optional Tools > Question sets dashboard uses the Flagship flag enable-decision-models-dashboard, exposed as enableDecisionModelsDashboard on the account profile. When dashboard access is disabled, use the programmatic interfaces below.

This flag controls only the dashboard. REST, SDK, MCP, and Code Mode model discovery, question-set configuration, and execution require no feature flag. Normal authentication, scopes, provider credentials, funding, and retention checks still apply.

What a question set holds

Each version of a question set holds these fields:

• questions: a map of question ID to question. It must contain at least one question.
• model: optional. The judgment model to request. When you leave it empty, calls use typesafe-ai/jev.
• inputMapping: optional. Runtype stores it with the version for your reference and doesn’t use it when a judgment runs.

The question ID is never sent to the model. The instructions and criteria carry the whole meaning of the question, so write them as if the ID did not exist.

The following table lists the three question types:

Runtype accepts noul, TypeSafe’s name for a boolean question, as an alias and stores the question as boolean. The instructions field and every criteria description accept a string, a JSON object, or a JSON array. A criteria description can also be null.

Give a choice question an option for no match, such as other, when the listed categories aren’t exhaustive. Questions share state but cannot read each other’s answers. Provider-specific limits can be stricter than the shared schema; inspect the selected model in the catalog.

Create and publish a question set

You manage question sets through the Runtype API, the Runtype MCP server, and the Code Mode MCP server. An API key needs the QUESTION_SETS:WRITE scope to create one, and the caller needs an organization context.

To create a question set and make it callable, follow these steps:

1. Create the question set. Its first version is a draft:

   cURL

   curl -X POST https://api.runtype.com/v1/question-sets \
     -H "Authorization: Bearer YOUR_API_KEY" \
     -H "Content-Type: application/json" \
     -d '{
       "slug": "ticket-triage",
       "name": "Ticket triage",
       "questions": {
         "route": {
           "type": "choice",
           "instructions": "Read the support ticket and choose the queue that handles it. Choose other when none of the named queues fit.",
           "criteria": {
             "billing": "Invoices, charges, refunds, payment methods, or plan pricing.",
             "technical": "Errors, outages, failed integrations, or unexpected product behavior.",
             "account": "Sign-in problems, seats, permissions, or organization settings.",
             "other": "Anything that none of the other labels describe."
           }
         },
         "requests_human": {
           "type": "boolean",
           "instructions": "Does the customer in the supplied ticket request a human support representative?",
           "criteria": {
             "true": "The customer explicitly asks to speak to a person or human support representative.",
             "false": "The customer makes no current request for a human. Quoting another person does not count."
           }
         }
       }
     }'

   Replace YOUR_API_KEY with a Runtype API key. The response carries questionSet.id, which starts with qset_, and version.versionNumber, which is 1.

2. Publish the version:

   cURL

   curl -X POST https://api.runtype.com/v1/question-sets/qset_YOUR_QUESTION_SET_ID/versions/1/publish \
     -H "Authorization: Bearer YOUR_API_KEY"

   Replace qset_YOUR_QUESTION_SET_ID with the questionSet.id from step 1. The published version becomes the current version, and it never changes again.

3. Reference the question set by its questionSetId from the judgment tool. For the configuration, see Configure a judgment instance.

The slug is fixed after creation and is unique within your organization. The slug takes lowercase letters, digits, -, and _, up to 64 characters, and must start with a lowercase letter or digit. You can’t delete a question set, so a slug stays taken. To rename a question set, change its name with PUT.

Change a question set

To change the questions, send PUT /v1/question-sets/{id} with new questions, inputMapping, or model. Runtype appends a new draft version and leaves every existing version unchanged. Fields that you leave out carry over from the most recent version. A PUT with only name or description updates the metadata and creates no version.

A draft does nothing until you publish it. Publishing makes that version the current version, so every reference without a pinned version picks up the published questions on its next call. To roll back, publish the earlier version again.

Thresholds stay with the caller

A question set is what you ask. A threshold, and the choice between act, abstain, and escalate, is what you do with the answer, so it belongs to the caller: a branch condition, a step’s configuration, or your code. Keeping the threshold out of the question set keeps the raw answers reusable. Move the threshold and the decision changes without a second call to the model.

The TypeSafe judgment tool

The builtin:typesafe_judge tool, named TypeSafe judgment in the catalog, runs a judgment and returns the answers. It works in an Agent’s tools, in a prompt step’s tools, in a Tool Call step, and directly over REST. The tool is read-only and has no approval gate by default. The approval settings that an author can set on any tool still apply.

The tool has two modes. The tool configuration decides which one runs.

Configure a judgment instance

A configured instance references a published question set. The Agent or step supplies only state. We recommend this mode: the questions live in one versioned place, and the model sees a tool that takes only state.

This prompt step configuration adds a judgment instance named triage:

{
  "tools": {
    "toolIds": ["builtin:typesafe_judge#triage"],
    "toolConfigs": {
      "builtin:typesafe_judge#triage": {
        "questionSetId": "qset_YOUR_QUESTION_SET_ID",
        "displayName": "Ticket triage"
      }
    }
  }
}

The text after # names the instance, so one Agent or step can carry several judgment instances with different question sets. The following table lists the instance settings:

The questions on a configured instance are locked to the referenced question set. A call that also sends questions fails with this error, so an Agent cannot rewrite the rubric:

This TypeSafe judgment is configured with question set "Ticket triage", so its questions cannot be replaced. Send only state.

The tool resolves the model in this order: the instance’s model, then the question set version’s model, then typesafe-ai/jev. The question set must resolve when the tool runs. A missing, unpublished, or foreign question set fails the call with an error that names its ID, and the tool never falls back to inline questions. Flow validation reports the same problem at save time. For the messages, see Tool configuration references an unresolved question set.

To call a judgment at a fixed point in a Flow, use a Tool Call step and pass the same settings in config.toolConfig:

{
  "type": "tool-call",
  "name": "Judge the ticket",
  "config": {
    "toolId": "builtin:typesafe_judge",
    "toolConfig": { "questionSetId": "qset_YOUR_QUESTION_SET_ID" },
    "parameters": { "state": "{{ticket}}" },
    "outputVariable": "judgment"
  }
}

A Tool Call step substitutes {{...}} references in a parameter as text. An object variable such as ticket therefore reaches the tool as its JSON text, which the judgment reads as text. The output variable holds the answers, so a condition reads {{judgment.route.choice}}. Replace qset_YOUR_QUESTION_SET_ID with your question set’s ID.

To act on a probability, add a conditional step after the Tool Call step and compare the answer with your calibrated threshold. The following conditions illustrate a two-sided boolean policy. They are not measured thresholds. A production policy needs labeled validation before it routes tickets automatically. The first path selects a human queue:

judgment.requests_human.probability >= 0.62

The second path sends an automated reply:

judgment.requests_human.probability <= 0.38

Otherwise runs when the answer falls between the two. Your policy abstains and sends the ticket to review. Handle refusals and errors before evaluating either condition.

Call a configured instance over REST

POST /v1/tools/{id}/execute runs the tool outside an Agent or Flow. Send the instance settings in toolConfig and the state in parameters. The route checks toolConfig against the tool’s configuration fields, so an unknown field or a wrong type fails with a 400 before any model call:

curl -X POST "https://api.runtype.com/v1/tools/builtin:typesafe_judge/execute" \
  -H "Authorization: Bearer YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "parameters": { "state": "My card was charged twice for the same order." },
    "toolConfig": { "questionSetId": "qset_YOUR_QUESTION_SET_ID", "version": 1 }
  }'

Replace qset_YOUR_QUESTION_SET_ID with your question set’s ID. The call resolves the question set version exactly as an Agent’s configured instance does. result holds the answers, and metadata.judgment reports how the judgment ran:

{
  "toolId": "builtin:typesafe_judge",
  "toolName": "TypeSafe judgment",
  "result": {
    "route": { "type": "choice", "choice": "billing" },
    "requests_human": { "type": "boolean", "probability": 0.18 }
  },
  "metadata": {
    "judgment": {
      "requestedModel": "typesafe-ai/jev",
      "resolvedModelId": null,
      "provider": "vercel",
      "route": "vercel-ai-gateway",
      "keySource": "platform",
      "questionSetId": "qset_YOUR_QUESTION_SET_ID",
      "questionSetVersion": 1,
      "calibrationId": null,
      "usage": { "inputTokens": 212, "outputTokens": 9, "reported": true },
      "estimatedCostUsd": 0.00031
    }
  }
}

The following table lists the metadata.judgment fields:

metadata is for the caller only. When an Agent, a prompt step, or a Tool Call step runs the tool, the model and the output variable receive the answers alone. The Runtype MCP execute_tool tool takes the same settings as tool_config, and the Code Mode method takes them as the third argument of executeTool.

The runtype CLI takes the settings with --config on runtype tools execute and runtype tools test:

runtype tools execute builtin:typesafe_judge \
  --input '{"state": "My card was charged twice for the same order."}' \
  --config '{"questionSetId": "qset_YOUR_QUESTION_SET_ID"}' \
  --json

The TypeScript SDK takes them as toolConfig. Narrow the response with isBuiltinToolExecuteResponse before you read metadata:

import { RuntypeClient, isBuiltinToolExecuteResponse } from '@runtypelabs/sdk'
const client = new RuntypeClient({ apiKey: process.env.RUNTYPE_API_KEY })
const response = await client.tools.execute('builtin:typesafe_judge', {
  toolId: 'builtin:typesafe_judge',
  parameters: { state: 'My card was charged twice for the same order.' },
  toolConfig: { questionSetId: 'qset_YOUR_QUESTION_SET_ID' },
})
if (isBuiltinToolExecuteResponse(response)) {
  console.log(response.result, response.metadata?.judgment)
}

Call the tool ad hoc

Without a questionSetId in the tool configuration, the Agent supplies questions beside state on every call, and can name a model. Use this mode for patterns such as “check this claim against this evidence”, where the question changes from call to call. Runtype checks ad hoc questions with the same rules as a question set and returns the rule’s sentence as a tool error, so the Agent can correct its next call.

To try ad hoc questions over REST before you save a question set, send questions beside state in parameters and leave out toolConfig:

curl -X POST "https://api.runtype.com/v1/tools/builtin:typesafe_judge/execute" \
  -H "Authorization: Bearer YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "parameters": {
      "state": "My card was charged twice for the same order.",
      "questions": {
        "is_refund": {
          "type": "boolean",
          "instructions": "Estimate the probability that the customer asks for a refund."
        }
      }
    }
  }'

The response’s metadata.judgment has no questionSetId or questionSetVersion for an ad hoc call.

A tool configuration can’t carry inline questions. Save the questions as a question set and reference it with questionSetId.

Read the answers

The tool returns normalized answers under their question IDs. A provider can return refusal for a question instead of its requested answer type. The following example shows the answers for the triage question set:

{
  "route": {
    "type": "choice",
    "choice": "billing",
    "probabilities": {
      "billing": 0.6,
      "technical": 0.3,
      "account": 0.08,
      "other": 0.02
    }
  },
  "requests_human": {
    "type": "boolean",
    "probability": 0.18
  }
}

The following table lists what each answer carries:

Every answer has a type. A refusal has this shape:

{ "type": "refusal" }

A refusal contains no synthetic probability or score. Treat it separately from a negative boolean, an authored other option, and a tool error.

The field names differ from provider APIs. Runtype uses boolean with probability, not noul or predicate. Optional providerConfidence preserves a provider-native statistic; Runtype doesn’t manufacture it. It isn’t selected-label probability or measured accuracy. Missing probabilities remain missing, so a probability-based policy cannot use that answer.

Credentials, limits, and errors

Credentials depend on the selected route. Jev through Vercel uses a gateway credential, not a TypeSafe API key. The OpenAI native route requires your organization’s OpenAI API key and doesn’t fall back to platform funding. Cloudflare uses the Runtype platform binding and funding checks. For key setup, see Connecting AI model providers.

A platform-funded judgment uses the model’s listed rates and the platform margin. Calls using your organization’s provider key are billed by that provider. The response reports available usage and estimated cost; missing price or usage information isn’t a zero-cost claim.

The typesafe-ai/jev model accepts up to 32,000 tokens for the state plus the longest question, and 64,000 tokens per request in total. A request over that budget fails, and the step or the Agent receives the error. Runtype doesn’t truncate the state for you, so send fewer fields.

Runtype retries a rate-limited or overloaded upstream. Each attempt times out after 5 seconds, Runtype makes up to three attempts, and the judgment fails if it has no answer after 15 seconds. A failed judgment fails the call the same way any tool failure does. Over REST, an error that you can fix returns a 4xx with a stable code. For the codes, see Tool execute error codes. Other failures, including a timeout, return a 500. A Tool Call step fails on a tool error unless you set onError. With onError set to continue, the output variable holds { "error": "...", "result": null } instead of answers, so add a step that checks for it before a branch reads the judgment.

Calibrate a question set version

A threshold is measured on your data, not chosen. A threshold fitted for one question set says nothing about another, and it can move when the judgment model changes. A calibration records that measurement against one published question set version.

To measure a question set version, follow these steps:

1. Collect a labeled set: states that represent your traffic, each with the answer that you expect for every question that you calibrate.
2. Run each state through the judgment tool with the version pinned. Keep each answer and the metadata.judgment.resolvedModelId. If that ID is null, independently verify the model revision before recording a calibration.
3. Split the labeled set into a training half and a held-out half. Keep related records together to avoid testing on near-duplicates of training examples.
4. On the train half, find the lowest decision-probability threshold at which the accuracy among handled answers reaches your target accuracy. An answer’s decision probability is the probability that it gives to the decision it makes.
5. Apply that threshold to the held-out half. If the held-out accuracy misses the target, the target is out of reach for that question, so don’t post a threshold for it.
6. On the held-out half, record correct and handled counts, handled rate, and expected calibration error (ECE). The fitter uses 10 equal-width probability bins.

To record a calibration, send the measurement for a published version. The numbers in this request illustrate the schema, not a recommended operating point:

curl -X POST https://api.runtype.com/v1/question-sets/qset_YOUR_QUESTION_SET_ID/versions/1/calibrations \
  -H "Authorization: Bearer YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "record": {
      "labeledSetRef": "support-tickets-2026-09",
      "perQuestion": {
        "requests_human": {
          "threshold": 0.62,
          "targetAccuracy": 0.95,
          "heldOutAccuracy": 0.951,
          "handledRate": 0.71,
          "ece": 0.03
        }
      },
      "measuredModel": "typesafe-ai/jev-1.13.0"
    }
  }'

The following table lists the fields of each perQuestion entry:

A stored calibration threshold measures decision probability: the probability that an answer gives to the decision it makes. It is never a position on a score’s levels. Each question type defines it as follows:

• Boolean: the probability of the side that the answer favors, max(probability, 1 - probability).
• Choice: the probability of the chosen label, probabilities[choice].
• Score: the probability mass of the most probable level in probabilities, not the score value.

An answer is handled when its decision probability is at or above the threshold. Refusals and Choice or Score answers without a distribution have no measurable decision probability. A threshold cannot turn them into accepted answers.

handledRate excludes those unmeasurable outcomes from its denominator. Report coverage over all original inputs separately, including refusals and failures; this conditional rate is not the fraction of all requests automated.

This fit measures classification accuracy among handled answers. It does not fit an asymmetric refund policy, a ranking objective, or the mean value of a Score. Evaluate those policies separately.

labeledSetRef names the labeled set that you measured on. Set measuredModel to the provider-reported revision of the runs that you measured, not to the typesafe-ai/jev alias. When a route doesn’t report a revision, metadata.judgment.resolvedModelId is null. Don’t treat a moving model alias as proof of a fixed revision. perQuestion needs at least one question, and every key must be a question that the version asks.

Optionally, add an evalRunId beside record. Runtype derives the calibration’s source from it, and never accepts source from the request:

• eval_suite: the evalRunId resolved to an eval run in your account.
• customer: the request had no evalRunId.

An evalRunId that doesn’t resolve fails the request with EVAL_RUN_NOT_FOUND. It is never recorded as customer instead.

Keep the fitter report with its sample counts. The stored calibration record contains summary rates, not the training and held-out counts. A result of 19 correct among 20 handled cases is not evidence of a guaranteed 95% success rate.

Each question-set version has one current calibration, not an active calibration for each provider. A model override doesn’t select a matching historical calibration. Keep separate question-set versions and measurements when comparing providers.

Calibrations are append-only. Each post adds a row and makes it the version’s current calibration, and no row is ever edited or deleted. A decision taken under an earlier calibration keeps the record that it used. List the history, newest first, with GET /v1/question-sets/{id}/versions/{versionNumber}/calibrations. The list takes limit, from 1 to 200 with a default of 50, and cursor.

A calibration is evidence, not a policy. Read the threshold from the calibration record and set it in your condition or code, as described in Thresholds stay with the caller.

When a provider reports a model version other than the calibration’s measuredModel, Runtype flags the calibration as stale: the judgment’s span in the execution trace carries runtype.judgment.calibrationStale. If the provider doesn’t report a revision, Runtype can’t verify the match and doesn’t set the flag. The flag is advisory, and the call still returns its answers unchanged. After a model version change, re-run the measurement and post a new calibration.

What an agent-requested judgment is not

The judgment tool locks the questions, not the evidence. An Agent still decides whether to call the tool and what to send as state. It can leave out a message that contradicts it, or send its own favorable summary.

A judgment that an Agent requests is therefore advisory. Use it for patterns such as “verify this before you answer”. Never use it as a safety gate, a completion check, a substitute for authorization, or a substitute for a required human approval.

Reference

Routes, MCP tools, and scopes

Question-set routes are under /v1/question-sets and require an organization context. Model discovery uses GET /v1/decision-models. The following table lists each route with its Runtype MCP tool, its Code Mode method, and its API key scope:

A key with QUESTION_SETS:WRITE can also read. get_question_set returns the versions with the question set. GET /v1/question-sets/{id}/resolved returns the question set exactly as an execution receives it: the current published version, or the published version that you name, with its current calibration. GET /v1/question-sets takes limit, from 1 to 200 with a default of 100, and cursor.

Error codes

A question set route returns an error body with an error message and, for most errors, a code. The following table lists the responses:

Tool execute error codes

POST /v1/tools/{id}/execute returns these codes for a judgment that you can fix. The same request fails the same way again, so a client should fix the request rather than retry it:

Flow validation reports a tool configuration that references a question set that can’t be resolved as QUESTION_SET_UNRESOLVED, which blocks the save. For its three messages, see Tool configuration references an unresolved question set.

Next steps

• Built-in tools: add the TypeSafe judgment tool to an Agent or a Flow.
• Using conditional steps: write the branch conditions that read an answer.
• Calling TypeSafe directly: call TypeSafe’s own API from a Make API Call step with a TypeSafe API key.
• Flow validation warnings: fix a reference to a question set that can’t be resolved.
• Setting up a Slack Surface: let a judgment decide when the bot replies and which Capability answers.