Rank, route, and verify in a Flow | Runtype Documentation

Build three Flows that take run-time input, ask a published question set, and turn the answers into a decision in code. The Flows rank candidates, route a support ticket with a review outcome, and hold a draft that its source doesn’t support.

Each Flow has the same three parts: a transform-data step that builds the state, a Tool Call step that asks the questions, and a transform-data step that applies your policy. For why each design works, see Designing judgments.

Before you begin

Use a Runtype API key with an organization context and the QUESTION_SETS:WRITE, FLOWS:WRITE, and FLOWS:EXECUTE scopes. Install curl, jq, and Bash.

Each run calls a model and can incur usage charges. Each recipe creates question sets and a Flow. A failed later step doesn’t delete resources that an earlier step created.

Set the API key and URL for the examples:

Environment

export RUNTYPE_API_KEY="YOUR_API_KEY"
export RUNTYPE_API_URL="https://api.runtype.com"

Replace YOUR_API_KEY with your Runtype API key.

Save these functions in your Bash session. They create and publish a question set, create a Flow, and run a Flow with inputs:

Reusable REST functions

set -euo pipefail
create_published_set() {
  local created id
  created=$(curl --fail-with-body --silent --show-error \
    "$RUNTYPE_API_URL/v1/question-sets" \
    -H "Authorization: Bearer $RUNTYPE_API_KEY" \
    -H "Content-Type: application/json" \
    --data @"$1")
  id=$(jq -er '.questionSet.id' <<< "$created")
  curl --fail-with-body --silent --show-error -X POST \
    "$RUNTYPE_API_URL/v1/question-sets/$id/versions/1/publish" \
    -H "Authorization: Bearer $RUNTYPE_API_KEY" > /dev/null
  printf '%s\n' "$id"
}
create_flow() {
  curl --fail-with-body --silent --show-error \
    "$RUNTYPE_API_URL/v1/flows" \
    -H "Authorization: Bearer $RUNTYPE_API_KEY" \
    -H "Content-Type: application/json" \
    --data @"$1" | jq -er '.id'
}
run_flow() {
  curl --fail-with-body --silent --show-error \
    "$RUNTYPE_API_URL/v1/dispatch" \
    -H "Authorization: Bearer $RUNTYPE_API_KEY" \
    -H "Content-Type: application/json" \
    --data "$(jq -n --arg id "$1" --slurpfile inputs "$2" \
      '{flow: {id: $id}, inputs: $inputs[0], options: {streamResponse: false}}')" |
    jq '.events[] | select(.type == "execution_complete") | .finalOutput | fromjson'
}

create_published_set prints the question set ID. run_flow prints the output variable of the last step.

Pass state to the Tool Call step as one template string, such as "{{rankInput.state}}". A Tool Call step substitutes variables only in top-level string parameters. If you write "state": { "message": "{{message}}" }, the Flow validates and runs, and the model judges the literal text {{message}}.

Rank candidates in one call

Put every candidate in one state and ask about each candidate slot. A candidate that the model judges alone has nothing to be compared with, so its score carries little ranking information.

This recipe asks two question sets about the same state so that you can compare them. The first has one score question per slot. The second has one choice question per ordered pair of slots.

To build and run the ranking Flow, follow these steps:

Generate both question sets. Each has eight slots, c1 to c8:

Create rank-scores.json and rank-pairs.json

jq -n '
  ["Does not satisfy the criterion at all.",
   "Satisfies the criterion weakly: far below what a typical candidate would.",
   "Satisfies the criterion substantially, with a clear shortfall.",
   "Satisfies the criterion about as fully as a candidate could."] as $levels |
  {slug: "example-rank-scores-8", name: "Rank up to 8 candidates by rubric score",
   questions: ([range(1; 9) | "c\(.)"] | map({key: ., value: {
     type: "score",
     instructions: "Rate how strongly candidate \(.) satisfies task.criterion.",
     criteria: $levels}}) | from_entries)}
' > rank-scores.json
jq -n '
  [range(1; 9) | "c\(.)"] as $slots |
  {slug: "example-rank-pairs-8", name: "Compare up to 8 candidates pairwise",
   questions: ([$slots[] as $a | $slots[] as $b | select($a != $b) | {
     key: "\($a)_\($b)", value: {
       type: "choice",
       instructions: "Compare candidates \($a) and \($b) on task.criterion. Which one satisfies it more?",
       criteria: {
         ($a): "Candidate \($a) satisfies task.criterion more than \($b).",
         ($b): "Candidate \($b) satisfies task.criterion more than \($a)."}}}] | from_entries)}
' > rank-pairs.json

The pair set holds 56 questions: each pair in both orders.

Create and publish both question sets:

Publish the ranking question sets

SCORES_SET_ID=$(create_published_set rank-scores.json)
PAIRS_SET_ID=$(create_published_set rank-pairs.json)

Save the script that builds the state. It fills unused slots with a marker that the ranking step ignores:

Create rank-assemble.js

cat > rank-assemble.js <<'JS'
const list = (input.candidates || []).slice(0, 8)
const shared = { task: { criterion: input.criterion } }
for (let i = 0; i < 8; i++) shared['c' + (i + 1)] = list[i] ? list[i].text : '(empty slot)'
return { state: JSON.stringify(shared), count: list.length }
JS

Save the script that ranks. It sorts by expected score, and separately by the summed probability that each candidate wins its pairs:

Create rank-in-code.js

cat > rank-in-code.js <<'JS'
const list = (input.candidates || []).slice(0, 8)
const n = list.length
const slot = (i) => 'c' + (i + 1)
const byScore = []
const unscored = []
for (let i = 0; i < n; i++) {
  const a = input.rankScores[slot(i)]
  if (a && a.type === 'score') byScore.push({ id: list[i].id, score: a.score, position: i })
  else unscored.push(list[i].id)
}
byScore.sort((x, y) => y.score - x.score || x.position - y.position)
const wins = Array(n).fill(0)
let comparable = true
for (let i = 0; i < n; i++) for (let j = 0; j < n; j++) {
  if (i === j) continue
  const first = input.rankPairs[slot(i) + '_' + slot(j)]
  const second = input.rankPairs[slot(j) + '_' + slot(i)]
  const a = first && first.probabilities ? first.probabilities[slot(i)] : undefined
  const b = second && second.probabilities ? second.probabilities[slot(i)] : undefined
  if (typeof a !== 'number' || typeof b !== 'number') { comparable = false; continue }
  wins[i] += (a + b) / 2
}
const byPairs = list.map((c, i) => ({ id: c.id, wins: wins[i], position: i })).sort((x, y) => y.wins - x.wins || x.position - y.position)
const round = (x) => Math.round(x * 100) / 100
return { byScore: byScore.map((r) => ({ id: r.id, score: round(r.score) })), byPairs: comparable ? byPairs.map((r) => ({ id: r.id, wins: round(r.wins) })) : null, unscored }
JS

A refused or missing score answer puts the candidate in unscored. The script never treats it as a zero.

Create the Flow:

Create the ranking Flow

jq -n --rawfile assemble rank-assemble.js --rawfile rank rank-in-code.js \
  --arg scores "$SCORES_SET_ID" --arg pairs "$PAIRS_SET_ID" '
  {name: "Rank candidates example", flowSteps: [
    {type: "transform-data", name: "Assemble ranking state", order: 0,
     config: {outputVariable: "rankInput", script: $assemble}},
    {type: "tool-call", name: "Score every candidate in one call", order: 1,
     config: {toolId: "builtin:decision_judge",
       toolConfig: {questionSetId: $scores, version: 1},
       parameters: {state: "{{rankInput.state}}"}, outputVariable: "rankScores"}},
    {type: "tool-call", name: "Compare every pair in one call", order: 2,
     config: {toolId: "builtin:decision_judge",
       toolConfig: {questionSetId: $pairs, version: 1},
       parameters: {state: "{{rankInput.state}}"}, outputVariable: "rankPairs"}},
    {type: "transform-data", name: "Rank in code", order: 3,
     config: {outputVariable: "ranking", script: $rank}}]}
' > rank-flow.json
RANK_FLOW_ID=$(create_flow rank-flow.json)

Save six candidates for one question, then run the Flow:

Run the ranking Flow

cat > rank-inputs.json <<'JSON'
{
  "criterion": "Relevance: the passage is more useful for answering the question \"How do I export my invoices as a PDF?\"",
  "candidates": [
    { "id": "kb-payment", "text": "To update your payment method, open Billing > Payment methods and click Replace card." },
    { "id": "kb-export-pdf", "text": "To export invoices as a PDF, open Billing > Invoices, select the invoices, click Export, and choose PDF." },
    { "id": "kb-hours", "text": "Our offices are closed on public holidays and reopen the next business day." },
    { "id": "kb-export-csv", "text": "Invoices can be exported as a CSV file from the Billing area." },
    { "id": "kb-billing-page", "text": "The Billing page lists your invoices, payment methods, and current plan." },
    { "id": "kb-export", "text": "To export invoices, open Billing > Invoices, select the invoices, and click Export." }
  ]
}
JSON
run_flow "$RANK_FLOW_ID" rank-inputs.json

A run of this Flow with typesafe-ai/jev returned the following output. Your numbers can differ:

Ranking output from one run

{
  "ranking": {
    "byScore": [
      { "id": "kb-export-pdf", "score": 3 },
      { "id": "kb-export", "score": 1.95 },
      { "id": "kb-export-csv", "score": 1.4 },
      { "id": "kb-billing-page", "score": 0.83 },
      { "id": "kb-payment", "score": 0.02 },
      { "id": "kb-hours", "score": 0 }
    ],
    "byPairs": [
      { "id": "kb-export-pdf", "wins": 5 },
      { "id": "kb-export", "wins": 3.95 },
      { "id": "kb-export-csv", "wins": 3.03 },
      { "id": "kb-billing-page", "wins": 2.02 },
      { "id": "kb-payment", "wins": 0.93 },
      { "id": "kb-hours", "wins": 0.08 }
    ],
    "unscored": []
  }
}

Both orderings agree here. Keep the score step when one call and the lowest cost matter most. Keep the pair step when candidates are close and a tie would change what you show. Remove the step that you don’t use, and its half of the ranking script.

A ranking always has a first place. To decide whether the first candidate is good enough to show, apply a minimum score that you measured, or ask a separate boolean question about the winner.

Route a ticket with a review outcome

Ask one choice question with an option for no match, and one boolean question for each queue that can apply together with another. The policy step routes a ticket only when one queue clearly leads.

To build and run the routing Flow, follow these steps:

Save the question set:

Create route-support.json

cat > route-support.json <<'JSON'
{
  "slug": "example-route-support",
  "name": "Route a support ticket with a review band",
  "questions": {
    "route": {
      "type": "choice",
      "instructions": "Choose the support queue for the request in ticket.message.",
      "criteria": {
        "billing": "Invoices, charges, refunds, or payment methods.",
        "technical": "Errors, outages, failed API calls, or unexpected product behavior.",
        "account": "Sign-in, password reset, seats, or permissions.",
        "other": "No named queue fits, or the message is not a support request."
      }
    },
    "needsBilling": {
      "type": "boolean",
      "instructions": "Does ticket.message ask for help with invoices, charges, refunds, or payment methods?",
      "criteria": {
        "true": "The customer currently needs billing help.",
        "false": "No current billing request. A past or quoted billing issue does not count."
      }
    },
    "needsTechnical": {
      "type": "boolean",
      "instructions": "Does ticket.message report an error, outage, failed API call, or unexpected product behavior?",
      "criteria": {
        "true": "The customer currently reports a technical problem.",
        "false": "No current technical problem is reported."
      }
    },
    "needsAccount": {
      "type": "boolean",
      "instructions": "Does ticket.message ask for help with sign-in, password reset, seats, or permissions?",
      "criteria": {
        "true": "The customer currently needs account-access help.",
        "false": "No current account-access request."
      }
    }
  }
}
JSON
ROUTE_SET_ID=$(create_published_set route-support.json)

Save the policy script. The value 0.8 is a starting point for the example, not a measured threshold:

Create route-policy.js

cat > route-policy.js <<'JS'
const actAt = 0.8
const j = input.ticketJudgment || {}
if (j.error) return { destination: 'review', reason: 'check_failed' }
const route = j.route
if (!route || route.type !== 'choice' || !route.probabilities) return { destination: 'review', reason: 'unanswered' }
const top = route.probabilities[route.choice]
const needs = { billing: j.needsBilling, technical: j.needsTechnical, account: j.needsAccount }
const answered = Object.keys(needs).every((q) => needs[q] && needs[q].type === 'boolean' && typeof needs[q].probability === 'number')
if (!answered) return { destination: 'review', reason: 'unanswered', leading: route.choice }
const queues = Object.keys(needs).filter((q) => needs[q].probability >= actAt)
if (!(top >= actAt)) return { destination: 'review', reason: 'no_clear_queue', leading: route.choice, top, alsoMentions: queues }
if (queues.length > 1) return { destination: 'review', reason: 'several_queues', leading: route.choice, alsoMentions: queues }
const expected = route.choice === 'other' ? [] : [route.choice]
if (queues.join() !== expected.join()) return { destination: 'review', reason: 'answers_disagree', leading: route.choice, top, alsoMentions: queues }
if (route.choice === 'other') return { destination: 'none', reason: 'no_queue_fits', top }
return { destination: route.choice, top }
JS

The script checks for a failed call and a refusal before it reads a probability. It sends the ticket to review unless all three queue checks returned an answer. It also sends the ticket to review when the queue checks don’t name the chosen queue and only that queue, or when the choice is other and a queue check is confident. It writes each comparison as !(value >= threshold) so that a missing probability goes to review.

Create the Flow. The Tool Call step sets onError to continue so that a failed judgment reaches the policy step as { "error": "...", "result": null }:

Create the routing Flow

jq -n --rawfile policy route-policy.js --arg set "$ROUTE_SET_ID" '
  {name: "Route a support ticket example", flowSteps: [
    {type: "transform-data", name: "Assemble ticket state", order: 0,
     config: {outputVariable: "routeInput",
       script: "return { state: JSON.stringify({ ticket: { message: String(input.message || \"\") } }) }"}},
    {type: "tool-call", name: "Judge the ticket", order: 1,
     config: {toolId: "builtin:decision_judge",
       toolConfig: {questionSetId: $set, version: 1},
       parameters: {state: "{{routeInput.state}}"},
       outputVariable: "ticketJudgment", onError: "continue"}},
    {type: "transform-data", name: "Apply routing policy", order: 2,
     config: {outputVariable: "routing", script: $policy}}]}
' > route-flow.json
ROUTE_FLOW_ID=$(create_flow route-flow.json)

Run the Flow on a ticket that names two queues:

Run the routing Flow

cat > route-inputs.json <<'JSON'
{ "message": "My invoice shows the wrong amount, and also the dashboard crashes when I open it." }
JSON
run_flow "$ROUTE_FLOW_ID" route-inputs.json

A run with typesafe-ai/jev sent this ticket to review:

Routing output from one run

{
  "routing": {
    "destination": "review",
    "reason": "no_clear_queue",
    "leading": "technical",
    "top": 0.68,
    "alsoMentions": ["billing", "technical"]
  }
}

In the same test, “Where can I download last month’s invoice?” returned billing with a top probability of 1. “I’d like to apply for the support engineer job posting.” returned none.

Replace 0.8 with a threshold that you fit on labeled tickets before you automate routing. See Calibrate a question set version.

Hold a draft that its source doesn’t support

Split the draft into statements in code and ask one choice question about each statement. The policy step holds the draft unless every statement is supported or makes no claim.

To build and run the verification Flow, follow these steps:

Generate and publish the question set. It has eight statement slots, k1 to k8:

Create and publish verify-claims.json

jq -n '
  {slug: "example-verify-claims-8", name: "Check up to 8 statements against a source",
   questions: ([range(1; 9) | "k\(.)"] | map({key: ., value: {
     type: "choice",
     instructions: "How does source bear on claims.\(.)? Use only source.",
     criteria: {
       supported: "Source establishes every material part of the claim.",
       contradicted: "Source states something incompatible with a material part of the claim.",
       unaddressed: "Source neither establishes nor contradicts the complete claim.",
       no_claim: "The sentence states no fact to check: a greeting, thanks, an apology, a sign-off, or an offer of further help."}}}) | from_entries)}
' > verify-claims.json
VERIFY_SET_ID=$(create_published_set verify-claims.json)

Save the script that splits the draft into sentences:

Create verify-split.js

cat > verify-split.js <<'JS'
const parts = String(input.draft || '').split(/(?<=[.!?])\s+/).map((s) => s.trim()).filter(Boolean)
const claims = {}
for (let i = 0; i < 8; i++) claims['k' + (i + 1)] = parts[i] || '(empty slot)'
return { state: JSON.stringify({ source: input.policyExcerpt, claims }), statements: parts.slice(0, 8), overflow: parts.length > 8 }
JS

Save the policy script. The value 0.9 is a starting point for the example, not a measured threshold:

Create verify-decide.js

cat > verify-decide.js <<'JS'
const minProbability = 0.9
const passing = ['supported', 'no_claim']
const checks = input.claimChecks || {}
const statements = input.verifyInput.statements
if (checks.error) return { action: 'hold', reason: 'check_failed', detail: String(checks.error) }
if (statements.length === 0) return { action: 'hold', reason: 'empty_draft' }
if (input.verifyInput.overflow) return { action: 'hold', reason: 'too_many_statements' }
const problems = []
let checked = 0
statements.forEach((text, i) => {
  const a = checks['k' + (i + 1)]
  if (!a || a.type !== 'choice' || !a.probabilities) problems.push({ statement: text, finding: 'unanswered' })
  else if (!passing.includes(a.choice)) problems.push({ statement: text, finding: a.choice })
  else if (!(a.probabilities[a.choice] >= minProbability)) problems.push({ statement: text, finding: 'uncertain' })
  else if (a.choice === 'supported') checked++
})
if (problems.length) return { action: 'hold', reason: 'unsupported_statements', problems }
return checked ? { action: 'send', checked } : { action: 'hold', reason: 'no_checkable_statement' }
JS

Every path that isn’t a clear pass returns hold: a failed call, an empty draft, more than eight statements, a refusal, a low probability, and a draft with no statement to check. The no_claim option lets a greeting or a sign-off pass, so a draft that opens with “Thanks for reaching out!” isn’t held for it.

Create the Flow:

Create the verification Flow

jq -n --rawfile split verify-split.js --rawfile decide verify-decide.js \
  --arg set "$VERIFY_SET_ID" '
  {name: "Verify a draft example", flowSteps: [
    {type: "transform-data", name: "Split the draft into statements", order: 0,
     config: {outputVariable: "verifyInput", script: $split}},
    {type: "tool-call", name: "Check each statement against the source", order: 1,
     config: {toolId: "builtin:decision_judge",
       toolConfig: {questionSetId: $set, version: 1},
       parameters: {state: "{{verifyInput.state}}"},
       outputVariable: "claimChecks", onError: "continue"}},
    {type: "transform-data", name: "Decide send or hold", order: 2,
     config: {outputVariable: "decision", script: $decide}}]}
' > verify-flow.json
VERIFY_FLOW_ID=$(create_flow verify-flow.json)

Run the Flow on a draft with one invented sentence among four supported sentences:

Run the verification Flow

cat > verify-inputs.json <<'JSON'
{
  "policyExcerpt": "Refund policy v3. Annual plans can be refunded in full within 14 days of purchase. After 14 days, annual plans are non-refundable. Monthly plans can be cancelled at any time and are not refunded for the current month. Refunds are issued to the original payment method within 5 business days. Customers must request refunds by email to billing@example.com.",
  "draft": "Annual plans are fully refundable during the first 14 days. Refunds go back to the original payment method. You will receive an email confirmation when the refund is issued. You have to email billing@example.com to ask for a refund. Monthly plans can be cancelled at any time."
}
JSON
run_flow "$VERIFY_FLOW_ID" verify-inputs.json

A run with typesafe-ai/jev held the draft and named the invented sentence:

Verification output from one run

{
  "decision": {
    "action": "hold",
    "reason": "unsupported_statements",
    "problems": [
      {
        "statement": "You will receive an email confirmation when the refund is issued.",
        "finding": "unaddressed"
      }
    ]
  }
}

In the same test, a draft of three supported sentences with a greeting and a sign-off returned { "action": "send", "checked": 3 }. A draft that said “Refunds take up to 30 business days.” returned hold with the finding contradicted.

A sentence split by punctuation is approximate. An abbreviation or a list can split in the wrong place, which produces a hold, not a wrong send. This check covers only what the excerpt says. It doesn’t establish that the excerpt is the right source or the current version.

Scripts that match these recipes

The repository file docs/examples/decision-models/patterns.mjs builds the same question sets, Flow steps, and inputs with Node.js 22 or later. Run node docs/examples/decision-models/patterns.mjs rank to print the requests without sending them.

Next steps

Designing judgments: choose question types, batch sizes, and thresholds.
Run judgment recipes: create and test three smaller question sets over REST.
Using tool-call steps: set parameters, timeouts, and error handling for the judgment step.
Using transform-data steps: write and test the state and policy scripts.