> For clean Markdown of any page, append .md to the page URL.
> For a complete documentation index, see https://docs.runtype.com/llms.txt.
> For AI client integration (Claude Code, Cursor, etc.), connect to the MCP server at https://docs.runtype.com/_mcp/server.

# Run judgment recipes

Create three question sets, test them against fictional evidence, and bind each published version to a Flow. The examples cover support routing, claim grounding, and editorial scoring. They don't send messages, issue refunds, or publish content.

## Before you begin

Use a Runtype API key with an organization context and the `QUESTION_SETS:WRITE`, `TOOLS:EXECUTE`, and `FLOWS:WRITE` scopes. Install `curl`, `jq`, and Bash.

These REST recipes require no feature flag. The Flagship flag `enable-decision-models-dashboard` controls only the optional Tools > Question sets dashboard, not model discovery, question-set authoring, or execution.

Each test calls a model and can incur usage charges. Each recipe creates a question set and a Flow. A failed later step doesn't delete resources that an earlier step created. Inspect the printed IDs before retrying.

Set the API key and model for the examples:

**`Environment`**

```bash title="Environment"
export RUNTYPE_API_KEY="YOUR_API_KEY"
export RUNTYPE_API_URL="https://api.runtype.com"
export DECISION_MODEL="typesafe-ai/jev"
```

Replace `YOUR_API_KEY` with your Runtype API key. Keep the key out of browser code, source control, and shared logs.

List supported models before changing `DECISION_MODEL`:

**`List decision models`**

```bash title="List decision models"
curl --fail-with-body --silent --show-error \
  "$RUNTYPE_API_URL/v1/decision-models" \
  -H "Authorization: Bearer $RUNTYPE_API_KEY" | jq '.models'
```

The catalog lists supported routes, limits, and credential requirements. It doesn't confirm that your organization can run a model. The test call checks credentials and policy.

## Define the create, publish, test, and bind commands

Save this function in your Bash session. It accepts a question-set JSON file and an evidence JSON file:

**`Reusable REST workflow`**

```bash title="Reusable REST workflow"
set -euo pipefail

publish_test_and_bind() {
  local definition="$1" evidence="$2"
  local created question_set_id version tested flow_payload

  created=$(curl --fail-with-body --silent --show-error \
    "$RUNTYPE_API_URL/v1/question-sets" \
    -H "Authorization: Bearer $RUNTYPE_API_KEY" \
    -H "Content-Type: application/json" \
    --data "$(jq --arg model "$DECISION_MODEL" \
      '. + {model: $model}' "$definition")")
  question_set_id=$(jq -er '.questionSet.id' <<< "$created")
  version=$(jq -er '.version.versionNumber' <<< "$created")
  printf 'Created %s version %s\n' "$question_set_id" "$version"

  curl --fail-with-body --silent --show-error -X POST \
    "$RUNTYPE_API_URL/v1/question-sets/$question_set_id/versions/$version/publish" \
    -H "Authorization: Bearer $RUNTYPE_API_KEY" > /dev/null

  tested=$(curl --fail-with-body --silent --show-error \
    "$RUNTYPE_API_URL/v1/tools/builtin:typesafe_judge/execute" \
    -H "Authorization: Bearer $RUNTYPE_API_KEY" \
    -H "Content-Type: application/json" \
    --data "$(jq -n --slurpfile state "$evidence" \
      --arg id "$question_set_id" --argjson version "$version" \
      '{parameters: {state: $state[0]},
        toolConfig: {questionSetId: $id, version: $version}}')")
  jq . <<< "$tested"
  jq -e --slurpfile definition "$definition" '
    .result as $answers |
    all($definition[0].questions | to_entries[];
      $answers[.key].type == .value.type)
  ' <<< "$tested" > /dev/null

  flow_payload=$(jq -n --slurpfile definition "$definition" \
    --slurpfile state "$evidence" \
    --arg id "$question_set_id" --argjson version "$version" '
    {name: ($definition[0].name + " example"), flowSteps: [{
      type: "tool-call", name: "Evaluate example evidence", order: 0,
      config: {
        toolId: "builtin:typesafe_judge",
        toolConfig: {questionSetId: $id, version: $version},
        parameters: {state: $state[0]}, outputVariable: "judgment"
      }
    }]}')
  curl --fail-with-body --silent --show-error \
    "$RUNTYPE_API_URL/v1/flows" \
    -H "Authorization: Bearer $RUNTYPE_API_KEY" \
    -H "Content-Type: application/json" \
    --data "$flow_payload" | jq '{id, name, flowSteps}'
}
```

The function creates a draft, publishes it, tests the configured tool, then creates a Flow with a pinned Tool Call step. A refusal or missing answer stops the function before it creates the Flow.

Each saved Flow contains the same fictional evidence used in the test. Replace that fixture with your application evidence before production use. `inputMapping` doesn't assemble evidence at runtime.

## Route a support request

Save a question set that separates queue selection from a request for a person:

**`Create support-triage.json`**

```bash title="Create support-triage.json"
cat > support-triage.json <<'JSON'
{
  "slug": "example-support-triage",
  "name": "Support triage",
  "questions": {
    "route": {
      "type": "choice",
      "instructions": "Choose the support queue for ticket.message.",
      "criteria": {
        "billing": "Invoices, duplicate charges, refunds, or payment methods.",
        "technical": "Errors, outages, integrations, or product malfunctions.",
        "account": "Sign-in, account recovery, seats, or permissions.",
        "other": "No named queue fits the request."
      }
    },
    "requestsHuman": {
      "type": "boolean",
      "instructions": "Does the customer in ticket.message ask to speak to a person?",
      "criteria": {
        "true": "The customer explicitly requests a human representative.",
        "false": "No human handoff is requested. A quotation alone does not count."
      }
    }
  }
}
JSON
```

Save a ticket with both a billing request and an explicit human-handoff request:

**`Create support-evidence.json`**

```bash title="Create support-evidence.json"
cat > support-evidence.json <<'JSON'
{
  "ticket": {
    "message": "Order A17 was charged twice. Can a person help me?"
  }
}
JSON
```

Create, publish, test, and bind the question set:

**`Run support triage`**

```bash title="Run support triage"
publish_test_and_bind support-triage.json support-evidence.json
```

Inspect `result.route` and `result.requestsHuman` in the printed test response. The intended labels are `billing` and `true`. A response might contain the following illustrative answers:

**`Illustrative support answers`**

```json title="Illustrative support answers"
{
  "route": {
    "type": "choice",
    "choice": "billing",
    "probabilities": {
      "billing": 0.94,
      "technical": 0.01,
      "account": 0.02,
      "other": 0.03
    }
  },
  "requestsHuman": { "type": "boolean", "probability": 0.98 }
}
```

Preserve both answers in the next step. Routing to billing must not discard the human request. A billing label doesn't authorize repayment.

Before automating routing, label mixed requests, out-of-scope tickets, quoted requests, and account-recovery cases. Measure wrong-queue and missed-handoff rates separately.

## Check an answer against its source

Save a question set that distinguishes supported, contradicted, and unaddressed claims:

**`Create claim-grounding.json`**

```bash title="Create claim-grounding.json"
cat > claim-grounding.json <<'JSON'
{
  "slug": "example-claim-grounding",
  "name": "Claim grounding",
  "questions": {
    "relation": {
      "type": "choice",
      "instructions": "Compare claim with sourceExcerpt only. Evaluate the complete claim, not shared wording.",
      "criteria": {
        "supported": "Every material part of the claim follows from the excerpt.",
        "contradicted": "The excerpt states something incompatible with the claim.",
        "unaddressed": "The excerpt neither establishes nor contradicts the complete claim."
      }
    }
  }
}
JSON
```

Save a claim that conflicts with the supplied policy:

**`Create claim-evidence.json`**

```bash title="Create claim-evidence.json"
cat > claim-evidence.json <<'JSON'
{
  "claim": "Annual subscriptions receive a full refund whenever they are cancelled.",
  "sourceExcerpt": "Annual subscriptions are non-refundable after 14 days. Requests during the first 14 days receive individual review.",
  "sourceVersion": "refund-policy-example-v1"
}
JSON
```

Create, publish, test, and bind the question set:

**`Run claim grounding`**

```bash title="Run claim grounding"
publish_test_and_bind claim-grounding.json claim-evidence.json
```

Inspect `result.relation`. The intended label is `contradicted`. An illustrative result is:

**`Illustrative grounding answer`**

```json title="Illustrative grounding answer"
{
  "relation": {
    "type": "choice",
    "choice": "contradicted",
    "probabilities": {
      "supported": 0.01,
      "contradicted": 0.96,
      "unaddressed": 0.03
    }
  }
}
```

Route this draft to correction or review. A missing source means not verified, not necessarily false. Check quote membership and source identity in code before the semantic check.

Test false acceptance using unsupported qualifiers, outdated policies, fabricated quotes, and instructions embedded in the source. The [TypeSafe citation cookbook](https://docs.typesafe.ai/cookbooks/citation_check) separates exact quote matching from contextual support.

## Score a draft on separate dimensions

Save a rubric for completeness and clarity:

**`Create article-rubric.json`**

```bash title="Create article-rubric.json"
cat > article-rubric.json <<'JSON'
{
  "slug": "example-article-rubric",
  "name": "Help article review",
  "questions": {
    "completeness": {
      "type": "score",
      "instructions": "Rate whether draft includes brief.requiredSteps. Ignore writing style.",
      "criteria": [
        "The draft does not explain the requested task.",
        "The draft addresses the task but omits a required action.",
        "Every required action appears, but a target or ordering is ambiguous.",
        "Every required action has an explicit target and executable order."
      ]
    },
    "clarity": {
      "type": "score",
      "instructions": "Rate whether brief.audience can follow the language in draft. Ignore missing actions.",
      "criteria": [
        "The language prevents the intended audience from following the draft.",
        "Unexplained terms or unclear references repeatedly obstruct reading.",
        "The instructions are readable except for one material ambiguity.",
        "The instructions use defined terms and unambiguous references."
      ]
    }
  }
}
JSON
```

Save a readable draft that omits the save action:

**`Create article-evidence.json`**

```bash title="Create article-evidence.json"
cat > article-evidence.json <<'JSON'
{
  "brief": {
    "audience": "A person using the settings page for the first time.",
    "requiredSteps": [
      "Open Settings.",
      "Open Notifications.",
      "Enter the new address.",
      "Click Save."
    ]
  },
  "draft": "Open Settings, then Notifications. Enter your new address in Notification email."
}
JSON
```

Create, publish, test, and bind the question set:

**`Run article scoring`**

```bash title="Run article scoring"
publish_test_and_bind article-rubric.json article-evidence.json
```

Inspect both dimensions. The completeness rubric assigns the omitted action to level `1`. Clarity can still be high. An illustrative response is:

**`Illustrative rubric answers`**

```json title="Illustrative rubric answers"
{
  "completeness": {
    "type": "score",
    "score": 1.1,
    "probabilities": { "0": 0.05, "1": 0.8, "2": 0.15, "3": 0 }
  },
  "clarity": {
    "type": "score",
    "score": 2.8,
    "probabilities": { "0": 0, "1": 0, "2": 0.2, "3": 0.8 }
  }
}
```

Keep incomplete drafts in the editing queue even when clarity is high. A weighted score can order that queue, but must not cancel a required completeness check.

Label the expected level on each dimension with your editors. Evaluate their agreement before fitting a model threshold. Compare the final editorial ordering separately from per-question accuracy.

## Evaluate before adding action branches

The examples establish request shapes and intended behavior. Their probabilities are illustrative, not measured results or recommended thresholds.

Save representative cases with a unique ID, state, and expected label for every question. Boolean labels are `true` or `false`; Choice labels are strings; Score labels are integer level indices.

A labeled support case has this shape:

**`Labeled case`**

```json title="Labeled case"
{
  "id": "duplicate-charge-human",
  "state": {
    "ticket": { "message": "Order A17 was charged twice. Can a person help?" }
  },
  "expected": { "route": "billing", "requestsHuman": true }
}
```

Use a separate held-out set to measure each fitted threshold. Record correct and handled counts, unavailable answers, refusals, and calls that fail. Don't infer reliability from one successful fixture.

Persist the tested question-set version and model identity with your results. Re-evaluate after changing the evidence builder, questions, provider, or action policy. See [calibration](/user-guide/flows/judgments#calibrate-a-question-set-version) for the record format and limits.

## Next steps

* [Designing judgments](/user-guide/flows/designing-judgments): improve evidence, answer choices, and rubric levels.
* [Using conditional steps](/user-guide/flows/using-conditional-steps): add branches after you validate an action policy.
* [Judgments](/user-guide/flows/judgments): inspect provider selection, response fields, and error handling.