> For clean Markdown of any page, append .md to the page URL. > For a complete documentation index, see https://docs.runtype.com/llms.txt. > For AI client integration (Claude Code, Cursor, etc.), connect to the MCP server at https://docs.runtype.com/_mcp/server. # Judgments > Ask typed questions about text or JSON and get back probabilities, labels, and scores that a Flow or Agent acts on. A judgment returns typed answers that your program can use: boolean probabilities, selected labels, or rubric scores. Some providers also return a refusal instead of an answer. Use a judgment for a bounded interpretation, such as a queue assignment or a claim check. Use a [prompt step](/user-guide/flows/using-prompt-steps) for prose, open-ended extraction, or work that needs tools. See [Designing judgments](/user-guide/flows/designing-judgments) for evidence and criteria, or [Run judgment recipes](/user-guide/flows/judgment-recipes) for complete REST examples. Judgments and prompt steps compose. Generate with a model, judge the result, then branch in the Flow. Or judge first to route cheaply, so that only the tickets that need a larger model reach one. Runtype supports decision models through Vercel AI Gateway, OpenAI, and Cloudflare. The default is TypeSafe Jev through Vercel AI Gateway. To call TypeSafe's own API instead, see [Calling TypeSafe directly](/user-guide/flows/calling-typesafe-directly). ## How a judgment differs from a prompt step A judgment separates questions from the code that acts on their answers. Its interface differs from a prompt step: * **There is no prompt.** You write a question set: named questions, each with a type, instructions, and criteria. Model settings such as temperature, a system prompt, and tools do not apply. * **The answer is data.** Runtype normalizes provider responses into shared answer types and preserves supplied probabilities. Store those answers to apply different thresholds without another model call. * **Costs depend on the route.** State, instructions, and criteria contribute input tokens. Providers can also charge for output. Read the catalog pricing and select the evidence that the questions require. * **A probability is not permission.** What a probability of 0.9 means on your data is something that you measure with labeled examples. For how Runtype stores that measurement, see [Calibrate a question set version](#calibrate-a-question-set-version). ### Keep the five quantities apart Conflating these quantities is how a wrong decision gets a number that looks trustworthy. Each one answers a different question: * **The boolean probability**: the estimate that a `boolean` question's statement is true. * **The selected label's probability**: the probability at `probabilities[choice]`, when the provider supplies a distribution. * **A distribution statistic**: a number that you compute from `probabilities`, such as the margin between the top two labels. It is absent when the answer carries no distribution. * **The rubric score**: the expectation over a `score` question's ordered levels, on the rubric's own scale. A 2.4 on a rubric of five levels is not a probability. * **The policy outcome**: the action that your thresholds turn an answer into. Plan for three outcomes from the start: act, abstain, and escalate. Boolean answers carry a probability, Choice answers carry a selected label, and Score answers carry a mean. Compute distribution statistics only when `probabilities` are available. The policy outcome is yours. For why, see [Thresholds stay with the caller](#thresholds-stay-with-the-caller). ## Decision models `GET /v1/decision-models` lists supported decision models, their limits, input kinds, pricing, and credential requirements. The equivalent MCP tool is `list_decision_models` with no arguments. Code Mode exposes `runtype.listDecisionModels()`. The catalog describes supported routes, not account readiness. A test call performs credential, funding, and policy checks. The following model IDs identify the native routes and the default gateway model: | Model ID | Route | Credentials and funding | | --------------------------- | --------------------- | ---------------------------------------------------------------------- | | `typesafe-ai/jev` | Vercel AI Gateway | Your organization's gateway key, or eligible Runtype platform funding. | | `openai/gpt-6-luna` | OpenAI Decisions | Your organization's OpenAI API key. No platform-funded fallback. | | `@cf/cloudflare/clef` | Cloudflare Workers AI | Runtype platform binding and eligible platform funding. | | `@cf/cloudflare/clef-flash` | Cloudflare Workers AI | Runtype platform binding and eligible platform funding. | | `@cf/cloudflare/clef-omni` | Cloudflare Workers AI | Runtype platform binding and eligible platform funding. | Other Vercel decision models can appear in the catalog. Select an ID from the response instead of using a chat-model alias. All Runtype judgment routes accept text, JSON objects, and JSON arrays. Top-level numbers, booleans, and `null` are not valid state. Native provider support for images or other media doesn't make those inputs available through this interface. A URL inside JSON is text, not an attached image. The OpenAI and Cloudflare native routes reject the strict zero-data-retention preference. They don't silently weaken that preference or switch providers. Use a compatible route or resolve your retention requirements before changing the setting. Cloudflare uses the ordinary Workers AI binding response. Per-call HTTP status, retry headers, and provider request IDs are not available on that route. The model gateway does not automatically retry an unknown binding failure; caller timeouts and their bounded retry policy still apply. ## Question sets A question set is the versioned rubric that your organization owns. You create it once, publish a version, and reference it by its `questionSetId` from every place that asks the same questions. One stored rubric keeps question definitions consistent across callers. Action rules and thresholds stay in caller code. ### Dashboard availability The optional **Tools > Question sets** dashboard uses the Flagship flag `enable-decision-models-dashboard`, exposed as `enableDecisionModelsDashboard` on the account profile. When dashboard access is disabled, use the programmatic interfaces below. This flag controls only the dashboard. REST, SDK, MCP, and Code Mode model discovery, question-set configuration, and execution require no feature flag. Normal authentication, scopes, provider credentials, funding, and retention checks still apply. ### What a question set holds Each version of a question set holds these fields: * **`questions`**: a map of question ID to question. It must contain at least one question. * **`model`**: optional. The judgment model to request. When you leave it empty, calls use `typesafe-ai/jev`. * **`inputMapping`**: optional. Runtype stores it with the version for your reference and doesn't use it when a judgment runs. The question ID is never sent to the model. The `instructions` and `criteria` carry the whole meaning of the question, so write them as if the ID did not exist. The following table lists the three question types: | Type | Ask it for | Criteria | | --------- | --------------------------------------------------------- | ------------------------------------------------------------------------------------ | | `boolean` | The probability that a statement about the state is true. | Optional. An object with `true` and `false` descriptions. | | `choice` | One label from a fixed set. | Required. An object of label to description, with 1 to 255 labels. | | `score` | An expected position on an ordered rubric. | Required. An ordered array of level descriptions, lowest first, with 2 to 10 levels. | Runtype accepts `noul`, TypeSafe's name for a `boolean` question, as an alias and stores the question as `boolean`. The `instructions` field and every criteria description accept a string, a JSON object, or a JSON array. A criteria description can also be `null`. Give a `choice` question an option for no match, such as `other`, when the listed categories aren't exhaustive. Questions share state but cannot read each other's answers. Provider-specific limits can be stricter than the shared schema; inspect the selected model in the catalog. ### Create and publish a question set You manage question sets through the Runtype API, the Runtype MCP server, and the [Code Mode MCP server](/developer-guides/integrations/runtype-mcp-server#connect-to-the-code-mode-server). An API key needs the `QUESTION_SETS:WRITE` scope to create one, and the caller needs an organization context. To create a question set and make it callable, follow these steps: 1. Create the question set. Its first version is a draft: **`cURL`** ```bash title="cURL" curl -X POST https://api.runtype.com/v1/question-sets \ -H "Authorization: Bearer YOUR_API_KEY" \ -H "Content-Type: application/json" \ -d '{ "slug": "ticket-triage", "name": "Ticket triage", "questions": { "route": { "type": "choice", "instructions": "Read the support ticket and choose the queue that handles it. Choose other when none of the named queues fit.", "criteria": { "billing": "Invoices, charges, refunds, payment methods, or plan pricing.", "technical": "Errors, outages, failed integrations, or unexpected product behavior.", "account": "Sign-in problems, seats, permissions, or organization settings.", "other": "Anything that none of the other labels describe." } }, "requests_human": { "type": "boolean", "instructions": "Does the customer in the supplied ticket request a human support representative?", "criteria": { "true": "The customer explicitly asks to speak to a person or human support representative.", "false": "The customer makes no current request for a human. Quoting another person does not count." } } } }' ``` Replace `YOUR_API_KEY` with a Runtype API key. The response carries `questionSet.id`, which starts with `qset_`, and `version.versionNumber`, which is `1`. 2. Publish the version: **`cURL`** ```bash title="cURL" curl -X POST https://api.runtype.com/v1/question-sets/qset_YOUR_QUESTION_SET_ID/versions/1/publish \ -H "Authorization: Bearer YOUR_API_KEY" ``` Replace `qset_YOUR_QUESTION_SET_ID` with the `questionSet.id` from step 1. The published version becomes the current version, and it never changes again. 3. Reference the question set by its `questionSetId` from the judgment tool. For the configuration, see [Configure a judgment instance](#configure-a-judgment-instance). The slug is fixed after creation and is unique within your organization. The slug takes lowercase letters, digits, `-`, and `_`, up to 64 characters, and must start with a lowercase letter or digit. You can't delete a question set, so a slug stays taken. To rename a question set, change its `name` with `PUT`. ### Change a question set To change the questions, send `PUT /v1/question-sets/{id}` with new `questions`, `inputMapping`, or `model`. Runtype appends a new draft version and leaves every existing version unchanged. Fields that you leave out carry over from the most recent version. A `PUT` with only `name` or `description` updates the metadata and creates no version. A draft does nothing until you publish it. Publishing makes that version the current version, so every reference without a pinned `version` picks up the published questions on its next call. To roll back, publish the earlier version again. ### Thresholds stay with the caller A question set is what you ask. A threshold, and the choice between act, abstain, and escalate, is what you do with the answer, so it belongs to the caller: a branch condition, a step's configuration, or your code. Keeping the threshold out of the question set keeps the raw answers reusable. Move the threshold and the decision changes without a second call to the model. ## The TypeSafe judgment tool The `builtin:typesafe_judge` tool, named **TypeSafe judgment** in the catalog, runs a judgment and returns the answers. It works in an Agent's tools, in a prompt step's tools, in a [Tool Call step](/user-guide/flows/flow-step-types-overview#tool-call), and directly over REST. The tool is read-only and has no approval gate by default. The approval settings that an author can set on any tool still apply. The tool has two modes. The tool configuration decides which one runs. ### Configure a judgment instance A configured instance references a published question set. The Agent or step supplies only `state`. We recommend this mode: the questions live in one versioned place, and the model sees a tool that takes only `state`. This prompt step configuration adds a judgment instance named `triage`: **`Prompt step tools`** ```json title="Prompt step tools" { "tools": { "toolIds": ["builtin:typesafe_judge#triage"], "toolConfigs": { "builtin:typesafe_judge#triage": { "questionSetId": "qset_YOUR_QUESTION_SET_ID", "displayName": "Ticket triage" } } } } ``` The text after `#` names the instance, so one Agent or step can carry several judgment instances with different question sets. The following table lists the instance settings: | Setting | Required | Description | | --------------- | -------- | ------------------------------------------------------------------------------------- | | `questionSetId` | Yes | The `qset_` ID of the question set to ask. | | `version` | No | A published version number. Leave it empty to use the current published version. | | `model` | No | The judgment model ID. Leave it empty to use the question set's model or the default. | | `displayName` | No | The name that the model sees, such as `Ticket triage`. | The questions on a configured instance are locked to the referenced question set. A call that also sends `questions` fails with this error, so an Agent cannot rewrite the rubric: ```text This TypeSafe judgment is configured with question set "Ticket triage", so its questions cannot be replaced. Send only state. ``` The tool resolves the model in this order: the instance's `model`, then the question set version's `model`, then `typesafe-ai/jev`. The question set must resolve when the tool runs. A missing, unpublished, or foreign question set fails the call with an error that names its ID, and the tool never falls back to inline questions. Flow validation reports the same problem at save time. For the messages, see [Tool configuration references an unresolved question set](/user-guide/flows/flow-validation-warnings#tool-configuration-references-an-unresolved-question-set). To call a judgment at a fixed point in a Flow, use a Tool Call step and pass the same settings in `config.toolConfig`: **`Tool Call step`** ```json title="Tool Call step" { "type": "tool-call", "name": "Judge the ticket", "config": { "toolId": "builtin:typesafe_judge", "toolConfig": { "questionSetId": "qset_YOUR_QUESTION_SET_ID" }, "parameters": { "state": "{{ticket}}" }, "outputVariable": "judgment" } } ``` A Tool Call step substitutes `{{...}}` references in a parameter as text. An object variable such as `ticket` therefore reaches the tool as its JSON text, which the judgment reads as text. The output variable holds the answers, so a condition reads `{{judgment.route.choice}}`. Replace `qset_YOUR_QUESTION_SET_ID` with your question set's ID. To act on a probability, add a [conditional step](/user-guide/flows/using-conditional-steps) after the Tool Call step and compare the answer with your calibrated threshold. The following conditions illustrate a two-sided boolean policy. They are not measured thresholds. A production policy needs labeled validation before it routes tickets automatically. The first path selects a human queue: ```plaintext judgment.requests_human.probability >= 0.62 ``` The second path sends an automated reply: ```plaintext judgment.requests_human.probability <= 0.38 ``` **Otherwise** runs when the answer falls between the two. Your policy abstains and sends the ticket to review. Handle refusals and errors before evaluating either condition. ### Call a configured instance over REST `POST /v1/tools/{id}/execute` runs the tool outside an Agent or Flow. Send the instance settings in `toolConfig` and the state in `parameters`. The route checks `toolConfig` against the tool's configuration fields, so an unknown field or a wrong type fails with a 400 before any model call: **`cURL`** ```bash title="cURL" curl -X POST "https://api.runtype.com/v1/tools/builtin:typesafe_judge/execute" \ -H "Authorization: Bearer YOUR_API_KEY" \ -H "Content-Type: application/json" \ -d '{ "parameters": { "state": "My card was charged twice for the same order." }, "toolConfig": { "questionSetId": "qset_YOUR_QUESTION_SET_ID", "version": 1 } }' ``` Replace `qset_YOUR_QUESTION_SET_ID` with your question set's ID. The call resolves the question set version exactly as an Agent's configured instance does. `result` holds the answers, and `metadata.judgment` reports how the judgment ran: **`Response`** ```json title="Response" { "toolId": "builtin:typesafe_judge", "toolName": "TypeSafe judgment", "result": { "route": { "type": "choice", "choice": "billing" }, "requests_human": { "type": "boolean", "probability": 0.18 } }, "metadata": { "judgment": { "requestedModel": "typesafe-ai/jev", "resolvedModelId": null, "provider": "vercel", "route": "vercel-ai-gateway", "keySource": "platform", "questionSetId": "qset_YOUR_QUESTION_SET_ID", "questionSetVersion": 1, "calibrationId": null, "usage": { "inputTokens": 212, "outputTokens": 9, "reported": true }, "estimatedCostUsd": 0.00031 } } } ``` The following table lists the `metadata.judgment` fields: | Field | Description | | ------------------------------------- | ------------------------------------------------------------------------------------------------- | | `requestedModel` | The model ID that the call asked for, such as the `typesafe-ai/jev` alias. | | `provider`, `route` | The provider and transport that served the judgment, not inferred from the requested alias. | | `resolvedModelId` | The provider-reported model identifier, which can still be an alias, or `null` when unreported. | | `keySource` | `platform` for Runtype platform funding, or `user` for your organization's provider key. | | `questionSetId`, `questionSetVersion` | The question set version that the call asked. Absent for an ad hoc call. | | `calibrationId` | The calibration attached to that version when it ran, or `null` when none was attached. | | `usage` | Input and output tokens. `reported` is `false` when the provider left the counts out. | | `estimatedCostUsd` | The cost at the model's listed per-token rates, before any platform margin. Absent when unpriced. | `metadata` is for the caller only. When an Agent, a prompt step, or a Tool Call step runs the tool, the model and the output variable receive the answers alone. The Runtype MCP `execute_tool` tool takes the same settings as `tool_config`, and the Code Mode method takes them as the third argument of `executeTool`. The `runtype` CLI takes the settings with `--config` on `runtype tools execute` and `runtype tools test`: **`CLI`** ```bash title="CLI" runtype tools execute builtin:typesafe_judge \ --input '{"state": "My card was charged twice for the same order."}' \ --config '{"questionSetId": "qset_YOUR_QUESTION_SET_ID"}' \ --json ``` The TypeScript SDK takes them as `toolConfig`. Narrow the response with `isBuiltinToolExecuteResponse` before you read `metadata`: **`TypeScript SDK`** ```typescript title="TypeScript SDK" import { RuntypeClient, isBuiltinToolExecuteResponse } from '@runtypelabs/sdk' const client = new RuntypeClient({ apiKey: process.env.RUNTYPE_API_KEY }) const response = await client.tools.execute('builtin:typesafe_judge', { toolId: 'builtin:typesafe_judge', parameters: { state: 'My card was charged twice for the same order.' }, toolConfig: { questionSetId: 'qset_YOUR_QUESTION_SET_ID' }, }) if (isBuiltinToolExecuteResponse(response)) { console.log(response.result, response.metadata?.judgment) } ``` ### Call the tool ad hoc Without a `questionSetId` in the tool configuration, the Agent supplies `questions` beside `state` on every call, and can name a `model`. Use this mode for patterns such as "check this claim against this evidence", where the question changes from call to call. Runtype checks ad hoc questions with the same rules as a question set and returns the rule's sentence as a tool error, so the Agent can correct its next call. To try ad hoc questions over REST before you save a question set, send `questions` beside `state` in `parameters` and leave out `toolConfig`: **`cURL`** ```bash title="cURL" curl -X POST "https://api.runtype.com/v1/tools/builtin:typesafe_judge/execute" \ -H "Authorization: Bearer YOUR_API_KEY" \ -H "Content-Type: application/json" \ -d '{ "parameters": { "state": "My card was charged twice for the same order.", "questions": { "is_refund": { "type": "boolean", "instructions": "Estimate the probability that the customer asks for a refund." } } } }' ``` The response's `metadata.judgment` has no `questionSetId` or `questionSetVersion` for an ad hoc call. A tool configuration can't carry inline questions. Save the questions as a question set and reference it with `questionSetId`. ### Read the answers The tool returns normalized answers under their question IDs. A provider can return `refusal` for a question instead of its requested answer type. The following example shows the answers for the triage question set: **`Tool result`** ```json title="Tool result" { "route": { "type": "choice", "choice": "billing", "probabilities": { "billing": 0.6, "technical": 0.3, "account": 0.08, "other": 0.02 } }, "requests_human": { "type": "boolean", "probability": 0.18 } } ``` The following table lists what each answer carries: | Question type | Answer fields | | ------------- | --------------------------------------------------------------------------------------------------- | | `boolean` | `probability`: the probability that the statement is true, from 0 to 1. | | `choice` | `choice`: the selected label. Optional `probabilities` and `providerConfidence`. | | `score` | `score`: the expectation over zero-based levels. Optional `probabilities` and `providerConfidence`. | Every answer has a `type`. A refusal has this shape: **`Refused answer`** ```json title="Refused answer" { "type": "refusal" } ``` A refusal contains no synthetic probability or score. Treat it separately from a negative boolean, an authored `other` option, and a tool error. The field names differ from provider APIs. Runtype uses `boolean` with `probability`, not `noul` or `predicate`. Optional `providerConfidence` preserves a provider-native statistic; Runtype doesn't manufacture it. It isn't selected-label probability or measured accuracy. Missing `probabilities` remain missing, so a probability-based policy cannot use that answer. ### Credentials, limits, and errors Credentials depend on the selected route. Jev through Vercel uses a gateway credential, not a TypeSafe API key. The OpenAI native route requires your organization's OpenAI API key and doesn't fall back to platform funding. Cloudflare uses the Runtype platform binding and funding checks. For key setup, see [Connecting AI model providers](/user-guide/integrations/connecting-ai-model-providers). A platform-funded judgment uses the model's listed rates and the platform margin. Calls using your organization's provider key are billed by that provider. The response reports available usage and estimated cost; missing price or usage information isn't a zero-cost claim. The `typesafe-ai/jev` model accepts up to 32,000 tokens for the state plus the longest question, and 64,000 tokens per request in total. A request over that budget fails, and the step or the Agent receives the error. Runtype doesn't truncate the state for you, so send fewer fields. Runtype retries a rate-limited or overloaded upstream. Each attempt times out after 5 seconds, Runtype makes up to three attempts, and the judgment fails if it has no answer after 15 seconds. A failed judgment fails the call the same way any tool failure does. Over REST, an error that you can fix returns a 4xx with a stable `code`. For the codes, see [Tool execute error codes](#tool-execute-error-codes). Other failures, including a timeout, return a 500. A Tool Call step fails on a tool error unless you set `onError`. With `onError` set to `continue`, the output variable holds `{ "error": "...", "result": null }` instead of answers, so add a step that checks for it before a branch reads the judgment. ## Calibrate a question set version A threshold is measured on your data, not chosen. A threshold fitted for one question set says nothing about another, and it can move when the judgment model changes. A calibration records that measurement against one published question set version. To measure a question set version, follow these steps: 1. Collect a labeled set: states that represent your traffic, each with the answer that you expect for every question that you calibrate. 2. Run each state through the judgment tool with the `version` pinned. Keep each answer and the `metadata.judgment.resolvedModelId`. If that ID is `null`, independently verify the model revision before recording a calibration. 3. Split the labeled set into a training half and a held-out half. Keep related records together to avoid testing on near-duplicates of training examples. 4. On the train half, find the lowest decision-probability threshold at which the accuracy among handled answers reaches your target accuracy. An answer's decision probability is the probability that it gives to the decision it makes. 5. Apply that threshold to the held-out half. If the held-out accuracy misses the target, the target is out of reach for that question, so don't post a threshold for it. 6. On the held-out half, record correct and handled counts, handled rate, and expected calibration error (ECE). The fitter uses 10 equal-width probability bins. To record a calibration, send the measurement for a published version. The numbers in this request illustrate the schema, not a recommended operating point: **`cURL`** ```bash title="cURL" curl -X POST https://api.runtype.com/v1/question-sets/qset_YOUR_QUESTION_SET_ID/versions/1/calibrations \ -H "Authorization: Bearer YOUR_API_KEY" \ -H "Content-Type: application/json" \ -d '{ "record": { "labeledSetRef": "support-tickets-2026-09", "perQuestion": { "requests_human": { "threshold": 0.62, "targetAccuracy": 0.95, "heldOutAccuracy": 0.951, "handledRate": 0.71, "ece": 0.03 } }, "measuredModel": "typesafe-ai/jev-1.13.0" } }' ``` The following table lists the fields of each `perQuestion` entry: | Field | Meaning | | ----------------- | -------------------------------------------------------------------------------- | | `threshold` | The fitted threshold on the answer's decision probability, from 0 to 1. | | `targetAccuracy` | The accuracy that the fit aimed for, from 0 to 1. | | `heldOutAccuracy` | The accuracy on the held-out half of the labeled set, from 0 to 1. | | `handledRate` | The share of measurable held-out answers accepted by the threshold, from 0 to 1. | | `ece` | The expected calibration error, from 0 to 1. | A stored calibration threshold measures decision probability: the probability that an answer gives to the decision it makes. It is never a position on a score's levels. Each question type defines it as follows: * **Boolean**: the probability of the side that the answer favors, `max(probability, 1 - probability)`. * **Choice**: the probability of the chosen label, `probabilities[choice]`. * **Score**: the probability mass of the most probable level in `probabilities`, not the `score` value. An answer is handled when its decision probability is at or above the threshold. Refusals and Choice or Score answers without a distribution have no measurable decision probability. A threshold cannot turn them into accepted answers. `handledRate` excludes those unmeasurable outcomes from its denominator. Report coverage over all original inputs separately, including refusals and failures; this conditional rate is not the fraction of all requests automated. This fit measures classification accuracy among handled answers. It does not fit an asymmetric refund policy, a ranking objective, or the mean value of a Score. Evaluate those policies separately. `labeledSetRef` names the labeled set that you measured on. Set `measuredModel` to the provider-reported revision of the runs that you measured, not to the `typesafe-ai/jev` alias. When a route doesn't report a revision, `metadata.judgment.resolvedModelId` is `null`. Don't treat a moving model alias as proof of a fixed revision. `perQuestion` needs at least one question, and every key must be a question that the version asks. Optionally, add an `evalRunId` beside `record`. Runtype derives the calibration's `source` from it, and never accepts `source` from the request: * **`eval_suite`**: the `evalRunId` resolved to an eval run in your account. * **`customer`**: the request had no `evalRunId`. An `evalRunId` that doesn't resolve fails the request with `EVAL_RUN_NOT_FOUND`. It is never recorded as `customer` instead. Keep the fitter report with its sample counts. The stored calibration record contains summary rates, not the training and held-out counts. A result of 19 correct among 20 handled cases is not evidence of a guaranteed 95% success rate. Each question-set version has one current calibration, not an active calibration for each provider. A model override doesn't select a matching historical calibration. Keep separate question-set versions and measurements when comparing providers. Calibrations are append-only. Each post adds a row and makes it the version's current calibration, and no row is ever edited or deleted. A decision taken under an earlier calibration keeps the record that it used. List the history, newest first, with `GET /v1/question-sets/{id}/versions/{versionNumber}/calibrations`. The list takes `limit`, from 1 to 200 with a default of 50, and `cursor`. A calibration is evidence, not a policy. Read the threshold from the calibration record and set it in your condition or code, as described in [Thresholds stay with the caller](#thresholds-stay-with-the-caller). When a provider reports a model version other than the calibration's `measuredModel`, Runtype flags the calibration as stale: the judgment's span in the execution trace carries `runtype.judgment.calibrationStale`. If the provider doesn't report a revision, Runtype can't verify the match and doesn't set the flag. The flag is advisory, and the call still returns its answers unchanged. After a model version change, re-run the measurement and post a new calibration. ## What an agent-requested judgment is not The judgment tool locks the questions, not the evidence. An Agent still decides whether to call the tool and what to send as `state`. It can leave out a message that contradicts it, or send its own favorable summary. A judgment that an Agent requests is therefore advisory. Use it for patterns such as "verify this before you answer". Never use it as a safety gate, a completion check, a substitute for authorization, or a substitute for a required human approval. ## Reference ### Routes, MCP tools, and scopes Question-set routes are under `/v1/question-sets` and require an organization context. Model discovery uses `GET /v1/decision-models`. The following table lists each route with its Runtype MCP tool, its Code Mode method, and its API key scope: | Route | MCP tool | Code Mode method | Scope | | ------------------------------------------------------- | -------------------------------- | ----------------------------- | --------------------- | | `GET /v1/question-sets` | `list_question_sets` | `listQuestionSets` | `QUESTION_SETS:READ` | | `POST /v1/question-sets` | `create_question_set` | `createQuestionSet` | `QUESTION_SETS:WRITE` | | `GET /v1/question-sets/{id}` | `get_question_set` | `getQuestionSet` | `QUESTION_SETS:READ` | | `PUT /v1/question-sets/{id}` | `update_question_set` | `updateQuestionSet` | `QUESTION_SETS:WRITE` | | `GET /v1/question-sets/{id}/versions` | none | `listQuestionSetVersions` | `QUESTION_SETS:READ` | | `POST /v1/question-sets/{id}/versions/{n}/publish` | `publish_question_set_version` | `publishQuestionSetVersion` | `QUESTION_SETS:WRITE` | | `POST /v1/question-sets/{id}/versions/{n}/calibrations` | `calibrate_question_set_version` | `calibrateQuestionSetVersion` | `QUESTION_SETS:WRITE` | | `GET /v1/question-sets/{id}/versions/{n}/calibrations` | `list_question_set_calibrations` | `listQuestionSetCalibrations` | `QUESTION_SETS:READ` | | `GET /v1/question-sets/{id}/resolved` | none | `resolveQuestionSet` | `QUESTION_SETS:READ` | A key with `QUESTION_SETS:WRITE` can also read. `get_question_set` returns the versions with the question set. `GET /v1/question-sets/{id}/resolved` returns the question set exactly as an execution receives it: the current published version, or the published `version` that you name, with its current calibration. `GET /v1/question-sets` takes `limit`, from 1 to 200 with a default of 100, and `cursor`. ### Error codes A question set route returns an error body with an `error` message and, for most errors, a `code`. The following table lists the responses: | Code | Status | Meaning | | ---------------------------------- | ------ | -------------------------------------------------------------------------------------------------------------------------------------------------------- | | `QUESTION_SET_INVALID` | 400 | A question breaks a rule, or a calibration measures a question that the version does not ask. `details` lists each rule's sentence and path. | | `QUESTION_SET_NOT_FOUND` | 404 | No question set with this ID exists for your account. | | none | 403 | The API key lacks the `QUESTION_SETS` scope that the route needs, or the caller has no organization context. Use an organization API key with the scope. | | `QUESTION_SET_VERSION_NOT_FOUND` | 404 | The question set has no version with this number. | | `QUESTION_SET_UNRESOLVED` | 404 | The resolve route found no question set with this ID, or no published version to return. | | `EVAL_RUN_NOT_FOUND` | 404 | The calibration's `evalRunId` doesn't resolve to an eval run in your account. | | `QUESTION_SET_VERSION_UNPUBLISHED` | 409 | The calibration targets a draft version. Publish the version first. | | `QUESTION_SET_SLUG_TAKEN` | 409 | Another question set in your organization uses this slug. | | `QUESTION_SET_CONCURRENT_EDIT` | 409 | The question set changed while your version was written. Retry the request. | ### Tool execute error codes `POST /v1/tools/{id}/execute` returns these codes for a judgment that you can fix. The same request fails the same way again, so a client should fix the request rather than retry it: | Code | Status | Meaning | | ---------------------------------------- | ------ | --------------------------------------------------------------------------------------------------------------------------------------------------------- | | `TOOL_CONFIG_INVALID` | 400 | `toolConfig` has an unknown field, a wrong type, or no `questionSetId`. | | `TOOL_CONFIG_NOT_SUPPORTED` | 400 | `toolConfig` was sent for a saved tool. Only `builtin:` and `platform:` tools take it. | | `JUDGMENT_STATE_INVALID` | 400 | `state` is missing, or is not text or JSON. | | `JUDGMENT_QUESTIONS_MISSING` | 400 | An ad hoc call sent no `questions`. | | `JUDGMENT_QUESTIONS_INVALID` | 400 | A question breaks a rule. `details.issues` lists each rule's sentence and path. | | `JUDGMENT_CONFIGURED_QUESTIONS_REJECTED` | 400 | A configured call also sent `questions`. | | `JUDGMENT_CONFIG_QUESTIONS_REJECTED` | 400 | The tool configuration carries `questions`. Save them as a question set and reference it with `questionSetId`. | | `JUDGMENT_QUESTION_SET_VERSION_INVALID` | 400 | The configured `version` is not a positive whole number, for example `1.5`. Use a published version number, or leave `version` empty for the current one. | | `JUDGMENT_REQUEST_INVALID` | 400 | Runtype rejected the request as malformed. | | `QUESTION_SET_UNRESOLVED` | 404 | The question set is missing, unpublished, or has no such published version. `details.reason` is `not_found`, `unpublished`, or `version_not_published`. | | `JUDGMENT_MODEL_NOT_SERVED` | 422 | The model is not admitted for the selected decision route or funding path. Inspect the decision-model catalog. | | `JUDGMENT_CREDENTIAL_UNAVAILABLE` | 422 | The selected route has no usable credential or funding path, or cannot honor the strict retention preference. | Flow validation reports a tool configuration that references a question set that can't be resolved as `QUESTION_SET_UNRESOLVED`, which blocks the save. For its three messages, see [Tool configuration references an unresolved question set](/user-guide/flows/flow-validation-warnings#tool-configuration-references-an-unresolved-question-set). ## Next steps * [Built-in tools](/user-guide/tools/built-in-tools#typesafe-judgment): add the TypeSafe judgment tool to an Agent or a Flow. * [Using conditional steps](/user-guide/flows/using-conditional-steps): write the branch conditions that read an answer. * [Calling TypeSafe directly](/user-guide/flows/calling-typesafe-directly): call TypeSafe's own API from a Make API Call step with a TypeSafe API key. * [Flow validation warnings](/user-guide/flows/flow-validation-warnings): fix a reference to a question set that can't be resolved. * [Setting up a Slack Surface](/user-guide/products-surfaces/setting-up-a-slack-surface#judge-mode): let a judgment decide when the bot replies and which Capability answers.