Skip to navigation
Flows

Run judgment recipes

Create three question sets, test them against fictional evidence, and bind each published version to a Flow. The examples cover support routing, claim grounding, and editorial scoring. They don’t send messages, issue refunds, or publish content.

Before you begin

Use a Runtype API key with an organization context and the QUESTION_SETS:WRITE, TOOLS:EXECUTE, and FLOWS:WRITE scopes. Install curl, jq, and Bash.

These REST recipes require no feature flag. The Flagship flag enable-decision-models-dashboard controls only the optional Tools > Question sets dashboard, not model discovery, question-set authoring, or execution.

Each test calls a model and can incur usage charges. Each recipe creates a question set and a Flow. A failed later step doesn’t delete resources that an earlier step created. Inspect the printed IDs before retrying.

Set the API key and model for the examples:

Environment
export RUNTYPE_API_KEY="YOUR_API_KEY"
export RUNTYPE_API_URL="https://api.runtype.com"
export DECISION_MODEL="typesafe-ai/jev"

Replace YOUR_API_KEY with your Runtype API key. Keep the key out of browser code, source control, and shared logs.

List supported models before changing DECISION_MODEL:

List decision models
curl --fail-with-body --silent --show-error \
  "$RUNTYPE_API_URL/v1/decision-models" \
  -H "Authorization: Bearer $RUNTYPE_API_KEY" | jq '.models'

The catalog lists supported routes, limits, and credential requirements. It doesn’t confirm that your organization can run a model. The test call checks credentials and policy.

Define the create, publish, test, and bind commands

Save this function in your Bash session. It accepts a question-set JSON file and an evidence JSON file:

Reusable REST workflow
set -euo pipefail
publish_test_and_bind() {
  local definition="$1" evidence="$2"
  local created question_set_id version tested flow_payload
  created=$(curl --fail-with-body --silent --show-error \
    "$RUNTYPE_API_URL/v1/question-sets" \
    -H "Authorization: Bearer $RUNTYPE_API_KEY" \
    -H "Content-Type: application/json" \
    --data "$(jq --arg model "$DECISION_MODEL" \
      '. + {model: $model}' "$definition")")
  question_set_id=$(jq -er '.questionSet.id' <<< "$created")
  version=$(jq -er '.version.versionNumber' <<< "$created")
  printf 'Created %s version %s\n' "$question_set_id" "$version"
  curl --fail-with-body --silent --show-error -X POST \
    "$RUNTYPE_API_URL/v1/question-sets/$question_set_id/versions/$version/publish" \
    -H "Authorization: Bearer $RUNTYPE_API_KEY" > /dev/null
  tested=$(curl --fail-with-body --silent --show-error \
    "$RUNTYPE_API_URL/v1/tools/builtin:typesafe_judge/execute" \
    -H "Authorization: Bearer $RUNTYPE_API_KEY" \
    -H "Content-Type: application/json" \
    --data "$(jq -n --slurpfile state "$evidence" \
      --arg id "$question_set_id" --argjson version "$version" \
      '{parameters: {state: $state[0]},
        toolConfig: {questionSetId: $id, version: $version}}')")
  jq . <<< "$tested"
  jq -e --slurpfile definition "$definition" '
    .result as $answers |
    all($definition[0].questions | to_entries[];
      $answers[.key].type == .value.type)
  ' <<< "$tested" > /dev/null
  flow_payload=$(jq -n --slurpfile definition "$definition" \
    --slurpfile state "$evidence" \
    --arg id "$question_set_id" --argjson version "$version" '
    {name: ($definition[0].name + " example"), flowSteps: [{
      type: "tool-call", name: "Evaluate example evidence", order: 0,
      config: {
        toolId: "builtin:typesafe_judge",
        toolConfig: {questionSetId: $id, version: $version},
        parameters: {state: $state[0]}, outputVariable: "judgment"
      }
    }]}')
  curl --fail-with-body --silent --show-error \
    "$RUNTYPE_API_URL/v1/flows" \
    -H "Authorization: Bearer $RUNTYPE_API_KEY" \
    -H "Content-Type: application/json" \
    --data "$flow_payload" | jq '{id, name, flowSteps}'
}

The function creates a draft, publishes it, tests the configured tool, then creates a Flow with a pinned Tool Call step. A refusal or missing answer stops the function before it creates the Flow.

Each saved Flow contains the same fictional evidence used in the test. Replace that fixture with your application evidence before production use. inputMapping doesn’t assemble evidence at runtime.

Route a support request

Save a question set that separates queue selection from a request for a person:

Create support-triage.json
cat > support-triage.json <<'JSON'
{
  "slug": "example-support-triage",
  "name": "Support triage",
  "questions": {
    "route": {
      "type": "choice",
      "instructions": "Choose the support queue for ticket.message.",
      "criteria": {
        "billing": "Invoices, duplicate charges, refunds, or payment methods.",
        "technical": "Errors, outages, integrations, or product malfunctions.",
        "account": "Sign-in, account recovery, seats, or permissions.",
        "other": "No named queue fits the request."
      }
    },
    "requestsHuman": {
      "type": "boolean",
      "instructions": "Does the customer in ticket.message ask to speak to a person?",
      "criteria": {
        "true": "The customer explicitly requests a human representative.",
        "false": "No human handoff is requested. A quotation alone does not count."
      }
    }
  }
}
JSON

Save a ticket with both a billing request and an explicit human-handoff request:

Create support-evidence.json
cat > support-evidence.json <<'JSON'
{
  "ticket": {
    "message": "Order A17 was charged twice. Can a person help me?"
  }
}
JSON

Create, publish, test, and bind the question set:

Run support triage
publish_test_and_bind support-triage.json support-evidence.json

Inspect result.route and result.requestsHuman in the printed test response. The intended labels are billing and true. A response might contain the following illustrative answers:

Illustrative support answers
{
  "route": {
    "type": "choice",
    "choice": "billing",
    "probabilities": {
      "billing": 0.94,
      "technical": 0.01,
      "account": 0.02,
      "other": 0.03
    }
  },
  "requestsHuman": { "type": "boolean", "probability": 0.98 }
}

Preserve both answers in the next step. Routing to billing must not discard the human request. A billing label doesn’t authorize repayment.

Before automating routing, label mixed requests, out-of-scope tickets, quoted requests, and account-recovery cases. Measure wrong-queue and missed-handoff rates separately.

Check an answer against its source

Save a question set that distinguishes supported, contradicted, and unaddressed claims:

Create claim-grounding.json
cat > claim-grounding.json <<'JSON'
{
  "slug": "example-claim-grounding",
  "name": "Claim grounding",
  "questions": {
    "relation": {
      "type": "choice",
      "instructions": "Compare claim with sourceExcerpt only. Evaluate the complete claim, not shared wording.",
      "criteria": {
        "supported": "Every material part of the claim follows from the excerpt.",
        "contradicted": "The excerpt states something incompatible with the claim.",
        "unaddressed": "The excerpt neither establishes nor contradicts the complete claim."
      }
    }
  }
}
JSON

Save a claim that conflicts with the supplied policy:

Create claim-evidence.json
cat > claim-evidence.json <<'JSON'
{
  "claim": "Annual subscriptions receive a full refund whenever they are cancelled.",
  "sourceExcerpt": "Annual subscriptions are non-refundable after 14 days. Requests during the first 14 days receive individual review.",
  "sourceVersion": "refund-policy-example-v1"
}
JSON

Create, publish, test, and bind the question set:

Run claim grounding
publish_test_and_bind claim-grounding.json claim-evidence.json

Inspect result.relation. The intended label is contradicted. An illustrative result is:

Illustrative grounding answer
{
  "relation": {
    "type": "choice",
    "choice": "contradicted",
    "probabilities": {
      "supported": 0.01,
      "contradicted": 0.96,
      "unaddressed": 0.03
    }
  }
}

Route this draft to correction or review. A missing source means not verified, not necessarily false. Check quote membership and source identity in code before the semantic check.

Test false acceptance using unsupported qualifiers, outdated policies, fabricated quotes, and instructions embedded in the source. The TypeSafe citation cookbook separates exact quote matching from contextual support.

Score a draft on separate dimensions

Save a rubric for completeness and clarity:

Create article-rubric.json
cat > article-rubric.json <<'JSON'
{
  "slug": "example-article-rubric",
  "name": "Help article review",
  "questions": {
    "completeness": {
      "type": "score",
      "instructions": "Rate whether draft includes brief.requiredSteps. Ignore writing style.",
      "criteria": [
        "The draft does not explain the requested task.",
        "The draft addresses the task but omits a required action.",
        "Every required action appears, but a target or ordering is ambiguous.",
        "Every required action has an explicit target and executable order."
      ]
    },
    "clarity": {
      "type": "score",
      "instructions": "Rate whether brief.audience can follow the language in draft. Ignore missing actions.",
      "criteria": [
        "The language prevents the intended audience from following the draft.",
        "Unexplained terms or unclear references repeatedly obstruct reading.",
        "The instructions are readable except for one material ambiguity.",
        "The instructions use defined terms and unambiguous references."
      ]
    }
  }
}
JSON

Save a readable draft that omits the save action:

Create article-evidence.json
cat > article-evidence.json <<'JSON'
{
  "brief": {
    "audience": "A person using the settings page for the first time.",
    "requiredSteps": [
      "Open Settings.",
      "Open Notifications.",
      "Enter the new address.",
      "Click Save."
    ]
  },
  "draft": "Open Settings, then Notifications. Enter your new address in Notification email."
}
JSON

Create, publish, test, and bind the question set:

Run article scoring
publish_test_and_bind article-rubric.json article-evidence.json

Inspect both dimensions. The completeness rubric assigns the omitted action to level 1. Clarity can still be high. An illustrative response is:

Illustrative rubric answers
{
  "completeness": {
    "type": "score",
    "score": 1.1,
    "probabilities": { "0": 0.05, "1": 0.8, "2": 0.15, "3": 0 }
  },
  "clarity": {
    "type": "score",
    "score": 2.8,
    "probabilities": { "0": 0, "1": 0, "2": 0.2, "3": 0.8 }
  }
}

Keep incomplete drafts in the editing queue even when clarity is high. A weighted score can order that queue, but must not cancel a required completeness check.

Label the expected level on each dimension with your editors. Evaluate their agreement before fitting a model threshold. Compare the final editorial ordering separately from per-question accuracy.

Evaluate before adding action branches

The examples establish request shapes and intended behavior. Their probabilities are illustrative, not measured results or recommended thresholds.

Save representative cases with a unique ID, state, and expected label for every question. Boolean labels are true or false; Choice labels are strings; Score labels are integer level indices.

A labeled support case has this shape:

Labeled case
{
  "id": "duplicate-charge-human",
  "state": {
    "ticket": { "message": "Order A17 was charged twice. Can a person help?" }
  },
  "expected": { "route": "billing", "requestsHuman": true }
}

Use a separate held-out set to measure each fitted threshold. Record correct and handled counts, unavailable answers, refusals, and calls that fail. Don’t infer reliability from one successful fixture.

Persist the tested question-set version and model identity with your results. Re-evaluate after changing the evidence builder, questions, provider, or action policy. See calibration for the record format and limits.

Next steps