> For clean Markdown of any page, append .md to the page URL. > For a complete documentation index, see https://docs.runtype.com/llms.txt. > For AI client integration (Claude Code, Cursor, etc.), connect to the MCP server at https://docs.runtype.com/_mcp/server. # Rank, route, and verify in a Flow > Three Flows that build judgment state at run time, ask a published question set, and apply a policy in code. Build three Flows that take run-time input, ask a published question set, and turn the answers into a decision in code. The Flows rank candidates, route a support ticket with a review outcome, and hold a draft that its source doesn't support. Each Flow has the same three parts: a transform-data step that builds the state, a Tool Call step that asks the questions, and a transform-data step that applies your policy. For why each design works, see [Designing judgments](/user-guide/flows/designing-judgments). ## Before you begin Use a Runtype API key with an organization context and the `QUESTION_SETS:WRITE`, `FLOWS:WRITE`, and `FLOWS:EXECUTE` scopes. Install `curl`, `jq`, and Bash. Each run calls a model and can incur usage charges. Each recipe creates question sets and a Flow. A failed later step doesn't delete resources that an earlier step created. Set the API key and URL for the examples: **`Environment`** ```bash title="Environment" export RUNTYPE_API_KEY="YOUR_API_KEY" export RUNTYPE_API_URL="https://api.runtype.com" ``` Replace `YOUR_API_KEY` with your Runtype API key. Save these functions in your Bash session. They create and publish a question set, create a Flow, and run a Flow with inputs: **`Reusable REST functions`** ```bash title="Reusable REST functions" set -euo pipefail create_published_set() { local created id created=$(curl --fail-with-body --silent --show-error \ "$RUNTYPE_API_URL/v1/question-sets" \ -H "Authorization: Bearer $RUNTYPE_API_KEY" \ -H "Content-Type: application/json" \ --data @"$1") id=$(jq -er '.questionSet.id' <<< "$created") curl --fail-with-body --silent --show-error -X POST \ "$RUNTYPE_API_URL/v1/question-sets/$id/versions/1/publish" \ -H "Authorization: Bearer $RUNTYPE_API_KEY" > /dev/null printf '%s\n' "$id" } create_flow() { curl --fail-with-body --silent --show-error \ "$RUNTYPE_API_URL/v1/flows" \ -H "Authorization: Bearer $RUNTYPE_API_KEY" \ -H "Content-Type: application/json" \ --data @"$1" | jq -er '.id' } run_flow() { curl --fail-with-body --silent --show-error \ "$RUNTYPE_API_URL/v1/dispatch" \ -H "Authorization: Bearer $RUNTYPE_API_KEY" \ -H "Content-Type: application/json" \ --data "$(jq -n --arg id "$1" --slurpfile inputs "$2" \ '{flow: {id: $id}, inputs: $inputs[0], options: {streamResponse: false}}')" | jq '.events[] | select(.type == "execution_complete") | .finalOutput | fromjson' } ``` `create_published_set` prints the question set ID. `run_flow` prints the output variable of the last step. > **Warning** > > Pass `state` to the Tool Call step as one template string, such as `"{{rankInput.state}}"`. A Tool Call step substitutes variables only in top-level string parameters. If you write `"state": { "message": "{{message}}" }`, the Flow validates and runs, and the model judges the literal text `{{message}}`. ## Rank candidates in one call Put every candidate in one state and ask about each candidate slot. A candidate that the model judges alone has nothing to be compared with, so its score carries little ranking information. This recipe asks two question sets about the same state so that you can compare them. The first has one `score` question per slot. The second has one `choice` question per ordered pair of slots. To build and run the ranking Flow, follow these steps: 1. Generate both question sets. Each has eight slots, `c1` to `c8`: **`Create rank-scores.json and rank-pairs.json`** ```bash title="Create rank-scores.json and rank-pairs.json" jq -n ' ["Does not satisfy the criterion at all.", "Satisfies the criterion weakly: far below what a typical candidate would.", "Satisfies the criterion substantially, with a clear shortfall.", "Satisfies the criterion about as fully as a candidate could."] as $levels | {slug: "example-rank-scores-8", name: "Rank up to 8 candidates by rubric score", questions: ([range(1; 9) | "c\(.)"] | map({key: ., value: { type: "score", instructions: "Rate how strongly candidate \(.) satisfies task.criterion.", criteria: $levels}}) | from_entries)} ' > rank-scores.json jq -n ' [range(1; 9) | "c\(.)"] as $slots | {slug: "example-rank-pairs-8", name: "Compare up to 8 candidates pairwise", questions: ([$slots[] as $a | $slots[] as $b | select($a != $b) | { key: "\($a)_\($b)", value: { type: "choice", instructions: "Compare candidates \($a) and \($b) on task.criterion. Which one satisfies it more?", criteria: { ($a): "Candidate \($a) satisfies task.criterion more than \($b).", ($b): "Candidate \($b) satisfies task.criterion more than \($a)."}}}] | from_entries)} ' > rank-pairs.json ``` The pair set holds 56 questions: each pair in both orders. 2. Create and publish both question sets: **`Publish the ranking question sets`** ```bash title="Publish the ranking question sets" SCORES_SET_ID=$(create_published_set rank-scores.json) PAIRS_SET_ID=$(create_published_set rank-pairs.json) ``` 3. Save the script that builds the state. It fills unused slots with a marker that the ranking step ignores: **`Create rank-assemble.js`** ```bash title="Create rank-assemble.js" cat > rank-assemble.js <<'JS' const list = (input.candidates || []).slice(0, 8) const shared = { task: { criterion: input.criterion } } for (let i = 0; i < 8; i++) shared['c' + (i + 1)] = list[i] ? list[i].text : '(empty slot)' return { state: JSON.stringify(shared), count: list.length } JS ``` 4. Save the script that ranks. It sorts by expected score, and separately by the summed probability that each candidate wins its pairs: **`Create rank-in-code.js`** ```bash title="Create rank-in-code.js" cat > rank-in-code.js <<'JS' const list = (input.candidates || []).slice(0, 8) const n = list.length const slot = (i) => 'c' + (i + 1) const byScore = [] const unscored = [] for (let i = 0; i < n; i++) { const a = input.rankScores[slot(i)] if (a && a.type === 'score') byScore.push({ id: list[i].id, score: a.score, position: i }) else unscored.push(list[i].id) } byScore.sort((x, y) => y.score - x.score || x.position - y.position) const wins = Array(n).fill(0) let comparable = true for (let i = 0; i < n; i++) for (let j = 0; j < n; j++) { if (i === j) continue const first = input.rankPairs[slot(i) + '_' + slot(j)] const second = input.rankPairs[slot(j) + '_' + slot(i)] const a = first && first.probabilities ? first.probabilities[slot(i)] : undefined const b = second && second.probabilities ? second.probabilities[slot(i)] : undefined if (typeof a !== 'number' || typeof b !== 'number') { comparable = false; continue } wins[i] += (a + b) / 2 } const byPairs = list.map((c, i) => ({ id: c.id, wins: wins[i], position: i })).sort((x, y) => y.wins - x.wins || x.position - y.position) const round = (x) => Math.round(x * 100) / 100 return { byScore: byScore.map((r) => ({ id: r.id, score: round(r.score) })), byPairs: comparable ? byPairs.map((r) => ({ id: r.id, wins: round(r.wins) })) : null, unscored } JS ``` A refused or missing score answer puts the candidate in `unscored`. The script never treats it as a zero. 5. Create the Flow: **`Create the ranking Flow`** ```bash title="Create the ranking Flow" jq -n --rawfile assemble rank-assemble.js --rawfile rank rank-in-code.js \ --arg scores "$SCORES_SET_ID" --arg pairs "$PAIRS_SET_ID" ' {name: "Rank candidates example", flowSteps: [ {type: "transform-data", name: "Assemble ranking state", order: 0, config: {outputVariable: "rankInput", script: $assemble}}, {type: "tool-call", name: "Score every candidate in one call", order: 1, config: {toolId: "builtin:decision_judge", toolConfig: {questionSetId: $scores, version: 1}, parameters: {state: "{{rankInput.state}}"}, outputVariable: "rankScores"}}, {type: "tool-call", name: "Compare every pair in one call", order: 2, config: {toolId: "builtin:decision_judge", toolConfig: {questionSetId: $pairs, version: 1}, parameters: {state: "{{rankInput.state}}"}, outputVariable: "rankPairs"}}, {type: "transform-data", name: "Rank in code", order: 3, config: {outputVariable: "ranking", script: $rank}}]} ' > rank-flow.json RANK_FLOW_ID=$(create_flow rank-flow.json) ``` 6. Save six candidates for one question, then run the Flow: **`Run the ranking Flow`** ```bash title="Run the ranking Flow" cat > rank-inputs.json <<'JSON' { "criterion": "Relevance: the passage is more useful for answering the question \"How do I export my invoices as a PDF?\"", "candidates": [ { "id": "kb-payment", "text": "To update your payment method, open Billing > Payment methods and click Replace card." }, { "id": "kb-export-pdf", "text": "To export invoices as a PDF, open Billing > Invoices, select the invoices, click Export, and choose PDF." }, { "id": "kb-hours", "text": "Our offices are closed on public holidays and reopen the next business day." }, { "id": "kb-export-csv", "text": "Invoices can be exported as a CSV file from the Billing area." }, { "id": "kb-billing-page", "text": "The Billing page lists your invoices, payment methods, and current plan." }, { "id": "kb-export", "text": "To export invoices, open Billing > Invoices, select the invoices, and click Export." } ] } JSON run_flow "$RANK_FLOW_ID" rank-inputs.json ``` A run of this Flow with `typesafe-ai/jev` returned the following output. Your numbers can differ: **`Ranking output from one run`** ```json title="Ranking output from one run" { "ranking": { "byScore": [ { "id": "kb-export-pdf", "score": 3 }, { "id": "kb-export", "score": 1.95 }, { "id": "kb-export-csv", "score": 1.4 }, { "id": "kb-billing-page", "score": 0.83 }, { "id": "kb-payment", "score": 0.02 }, { "id": "kb-hours", "score": 0 } ], "byPairs": [ { "id": "kb-export-pdf", "wins": 5 }, { "id": "kb-export", "wins": 3.95 }, { "id": "kb-export-csv", "wins": 3.03 }, { "id": "kb-billing-page", "wins": 2.02 }, { "id": "kb-payment", "wins": 0.93 }, { "id": "kb-hours", "wins": 0.08 } ], "unscored": [] } } ``` Both orderings agree here. Keep the score step when one call and the lowest cost matter most. Keep the pair step when candidates are close and a tie would change what you show. Remove the step that you don't use, and its half of the ranking script. A ranking always has a first place. To decide whether the first candidate is good enough to show, apply a minimum score that you measured, or ask a separate `boolean` question about the winner. ## Route a ticket with a review outcome Ask one `choice` question with an option for no match, and one `boolean` question for each queue that can apply together with another. The policy step routes a ticket only when one queue clearly leads. To build and run the routing Flow, follow these steps: 1. Save the question set: **`Create route-support.json`** ```bash title="Create route-support.json" cat > route-support.json <<'JSON' { "slug": "example-route-support", "name": "Route a support ticket with a review band", "questions": { "route": { "type": "choice", "instructions": "Choose the support queue for the request in ticket.message.", "criteria": { "billing": "Invoices, charges, refunds, or payment methods.", "technical": "Errors, outages, failed API calls, or unexpected product behavior.", "account": "Sign-in, password reset, seats, or permissions.", "other": "No named queue fits, or the message is not a support request." } }, "needsBilling": { "type": "boolean", "instructions": "Does ticket.message ask for help with invoices, charges, refunds, or payment methods?", "criteria": { "true": "The customer currently needs billing help.", "false": "No current billing request. A past or quoted billing issue does not count." } }, "needsTechnical": { "type": "boolean", "instructions": "Does ticket.message report an error, outage, failed API call, or unexpected product behavior?", "criteria": { "true": "The customer currently reports a technical problem.", "false": "No current technical problem is reported." } }, "needsAccount": { "type": "boolean", "instructions": "Does ticket.message ask for help with sign-in, password reset, seats, or permissions?", "criteria": { "true": "The customer currently needs account-access help.", "false": "No current account-access request." } } } } JSON ROUTE_SET_ID=$(create_published_set route-support.json) ``` 2. Save the policy script. The value `0.8` is a starting point for the example, not a measured threshold: **`Create route-policy.js`** ```bash title="Create route-policy.js" cat > route-policy.js <<'JS' const actAt = 0.8 const j = input.ticketJudgment || {} if (j.error) return { destination: 'review', reason: 'check_failed' } const route = j.route if (!route || route.type !== 'choice' || !route.probabilities) return { destination: 'review', reason: 'unanswered' } const top = route.probabilities[route.choice] const needs = { billing: j.needsBilling, technical: j.needsTechnical, account: j.needsAccount } const answered = Object.keys(needs).every((q) => needs[q] && needs[q].type === 'boolean' && typeof needs[q].probability === 'number') if (!answered) return { destination: 'review', reason: 'unanswered', leading: route.choice } const queues = Object.keys(needs).filter((q) => needs[q].probability >= actAt) if (!(top >= actAt)) return { destination: 'review', reason: 'no_clear_queue', leading: route.choice, top, alsoMentions: queues } if (queues.length > 1) return { destination: 'review', reason: 'several_queues', leading: route.choice, alsoMentions: queues } const expected = route.choice === 'other' ? [] : [route.choice] if (queues.join() !== expected.join()) return { destination: 'review', reason: 'answers_disagree', leading: route.choice, top, alsoMentions: queues } if (route.choice === 'other') return { destination: 'none', reason: 'no_queue_fits', top } return { destination: route.choice, top } JS ``` The script checks for a failed call and a refusal before it reads a probability. It sends the ticket to review unless all three queue checks returned an answer. It also sends the ticket to review when the queue checks don't name the chosen queue and only that queue, or when the choice is `other` and a queue check is confident. It writes each comparison as `!(value >= threshold)` so that a missing probability goes to review. 3. Create the Flow. The Tool Call step sets `onError` to `continue` so that a failed judgment reaches the policy step as `{ "error": "...", "result": null }`: **`Create the routing Flow`** ```bash title="Create the routing Flow" jq -n --rawfile policy route-policy.js --arg set "$ROUTE_SET_ID" ' {name: "Route a support ticket example", flowSteps: [ {type: "transform-data", name: "Assemble ticket state", order: 0, config: {outputVariable: "routeInput", script: "return { state: JSON.stringify({ ticket: { message: String(input.message || \"\") } }) }"}}, {type: "tool-call", name: "Judge the ticket", order: 1, config: {toolId: "builtin:decision_judge", toolConfig: {questionSetId: $set, version: 1}, parameters: {state: "{{routeInput.state}}"}, outputVariable: "ticketJudgment", onError: "continue"}}, {type: "transform-data", name: "Apply routing policy", order: 2, config: {outputVariable: "routing", script: $policy}}]} ' > route-flow.json ROUTE_FLOW_ID=$(create_flow route-flow.json) ``` 4. Run the Flow on a ticket that names two queues: **`Run the routing Flow`** ```bash title="Run the routing Flow" cat > route-inputs.json <<'JSON' { "message": "My invoice shows the wrong amount, and also the dashboard crashes when I open it." } JSON run_flow "$ROUTE_FLOW_ID" route-inputs.json ``` A run with `typesafe-ai/jev` sent this ticket to review: **`Routing output from one run`** ```json title="Routing output from one run" { "routing": { "destination": "review", "reason": "no_clear_queue", "leading": "technical", "top": 0.68, "alsoMentions": ["billing", "technical"] } } ``` In the same test, "Where can I download last month's invoice?" returned `billing` with a top probability of `1`. "I'd like to apply for the support engineer job posting." returned `none`. Replace `0.8` with a threshold that you fit on labeled tickets before you automate routing. See [Calibrate a question set version](/user-guide/flows/judgments#calibrate-a-question-set-version). ## Hold a draft that its source doesn't support Split the draft into statements in code and ask one `choice` question about each statement. The policy step holds the draft unless every statement is `supported` or makes no claim. To build and run the verification Flow, follow these steps: 1. Generate and publish the question set. It has eight statement slots, `k1` to `k8`: **`Create and publish verify-claims.json`** ```bash title="Create and publish verify-claims.json" jq -n ' {slug: "example-verify-claims-8", name: "Check up to 8 statements against a source", questions: ([range(1; 9) | "k\(.)"] | map({key: ., value: { type: "choice", instructions: "How does source bear on claims.\(.)? Use only source.", criteria: { supported: "Source establishes every material part of the claim.", contradicted: "Source states something incompatible with a material part of the claim.", unaddressed: "Source neither establishes nor contradicts the complete claim.", no_claim: "The sentence states no fact to check: a greeting, thanks, an apology, a sign-off, or an offer of further help."}}}) | from_entries)} ' > verify-claims.json VERIFY_SET_ID=$(create_published_set verify-claims.json) ``` 2. Save the script that splits the draft into sentences: **`Create verify-split.js`** ```bash title="Create verify-split.js" cat > verify-split.js <<'JS' const parts = String(input.draft || '').split(/(?<=[.!?])\s+/).map((s) => s.trim()).filter(Boolean) const claims = {} for (let i = 0; i < 8; i++) claims['k' + (i + 1)] = parts[i] || '(empty slot)' return { state: JSON.stringify({ source: input.policyExcerpt, claims }), statements: parts.slice(0, 8), overflow: parts.length > 8 } JS ``` 3. Save the policy script. The value `0.9` is a starting point for the example, not a measured threshold: **`Create verify-decide.js`** ```bash title="Create verify-decide.js" cat > verify-decide.js <<'JS' const minProbability = 0.9 const passing = ['supported', 'no_claim'] const checks = input.claimChecks || {} const statements = input.verifyInput.statements if (checks.error) return { action: 'hold', reason: 'check_failed', detail: String(checks.error) } if (statements.length === 0) return { action: 'hold', reason: 'empty_draft' } if (input.verifyInput.overflow) return { action: 'hold', reason: 'too_many_statements' } const problems = [] let checked = 0 statements.forEach((text, i) => { const a = checks['k' + (i + 1)] if (!a || a.type !== 'choice' || !a.probabilities) problems.push({ statement: text, finding: 'unanswered' }) else if (!passing.includes(a.choice)) problems.push({ statement: text, finding: a.choice }) else if (!(a.probabilities[a.choice] >= minProbability)) problems.push({ statement: text, finding: 'uncertain' }) else if (a.choice === 'supported') checked++ }) if (problems.length) return { action: 'hold', reason: 'unsupported_statements', problems } return checked ? { action: 'send', checked } : { action: 'hold', reason: 'no_checkable_statement' } JS ``` Every path that isn't a clear pass returns `hold`: a failed call, an empty draft, more than eight statements, a refusal, a low probability, and a draft with no statement to check. The `no_claim` option lets a greeting or a sign-off pass, so a draft that opens with "Thanks for reaching out!" isn't held for it. 4. Create the Flow: **`Create the verification Flow`** ```bash title="Create the verification Flow" jq -n --rawfile split verify-split.js --rawfile decide verify-decide.js \ --arg set "$VERIFY_SET_ID" ' {name: "Verify a draft example", flowSteps: [ {type: "transform-data", name: "Split the draft into statements", order: 0, config: {outputVariable: "verifyInput", script: $split}}, {type: "tool-call", name: "Check each statement against the source", order: 1, config: {toolId: "builtin:decision_judge", toolConfig: {questionSetId: $set, version: 1}, parameters: {state: "{{verifyInput.state}}"}, outputVariable: "claimChecks", onError: "continue"}}, {type: "transform-data", name: "Decide send or hold", order: 2, config: {outputVariable: "decision", script: $decide}}]} ' > verify-flow.json VERIFY_FLOW_ID=$(create_flow verify-flow.json) ``` 5. Run the Flow on a draft with one invented sentence among four supported sentences: **`Run the verification Flow`** ```bash title="Run the verification Flow" cat > verify-inputs.json <<'JSON' { "policyExcerpt": "Refund policy v3. Annual plans can be refunded in full within 14 days of purchase. After 14 days, annual plans are non-refundable. Monthly plans can be cancelled at any time and are not refunded for the current month. Refunds are issued to the original payment method within 5 business days. Customers must request refunds by email to billing@example.com.", "draft": "Annual plans are fully refundable during the first 14 days. Refunds go back to the original payment method. You will receive an email confirmation when the refund is issued. You have to email billing@example.com to ask for a refund. Monthly plans can be cancelled at any time." } JSON run_flow "$VERIFY_FLOW_ID" verify-inputs.json ``` A run with `typesafe-ai/jev` held the draft and named the invented sentence: **`Verification output from one run`** ```json title="Verification output from one run" { "decision": { "action": "hold", "reason": "unsupported_statements", "problems": [ { "statement": "You will receive an email confirmation when the refund is issued.", "finding": "unaddressed" } ] } } ``` In the same test, a draft of three supported sentences with a greeting and a sign-off returned `{ "action": "send", "checked": 3 }`. A draft that said "Refunds take up to 30 business days." returned `hold` with the finding `contradicted`. A sentence split by punctuation is approximate. An abbreviation or a list can split in the wrong place, which produces a hold, not a wrong send. This check covers only what the excerpt says. It doesn't establish that the excerpt is the right source or the current version. ## Scripts that match these recipes The repository file `docs/examples/decision-models/patterns.mjs` builds the same question sets, Flow steps, and inputs with Node.js 22 or later. Run `node docs/examples/decision-models/patterns.mjs rank` to print the requests without sending them. ## Next steps * [Designing judgments](/user-guide/flows/designing-judgments): choose question types, batch sizes, and thresholds. * [Run judgment recipes](/user-guide/flows/judgment-recipes): create and test three smaller question sets over REST. * [Using tool-call steps](/user-guide/flows/using-tool-call-steps): set parameters, timeouts, and error handling for the judgment step. * [Using transform-data steps](/user-guide/flows/using-transform-data-steps): write and test the state and policy scripts.