Read signals out of every agent turn

Trace signals are typed fields that Runtype reads out of every completed turn of an Agent, such as “did the assistant promise a follow-up?” or “how frustrated is the user?”. This guide shows you how to create a signal set, preview it on recent traces, bind it to an Agent, and read the results on runs and traces.

You configure signals through the Runtype Model Context Protocol (MCP) server, the REST API, the SDK, or the CLI. The dashboard only displays them.

Every organization also gets the built-in Failure facts set without configuring anything. See Failure facts on every Agent.

Before you begin

You need the following:

• A Runtype Agent, native or external, with completed executions in the last 30 days. An external Agent sends its traces to Runtype over OpenTelemetry. For details, see Reporting telemetry from an external agent.
• For the REST API, SDK, or CLI, an API key with the SIGNALS:WRITE permission. Reading results and stats needs SIGNALS:READ. For details, see Authentication.

How signals are evaluated

A signal set lists up to 40 signals. Each signal has a key, a type, and instructions. Runtype sends each turn’s signals to two kinds of model in one call per kind:

Each answer falls into a band. An answer at or above bands.act (default 0.9) is in the act band. An answer between bands.review (default 0.6) and bands.act is in the review band: count it, but don’t act on it without a person checking it.

A signal fires when a bool is true, or a choice value is listed in notable, in the act band. Extraction signals carry values and never fire on their own.

Failure facts on every Agent

Runtype computes a built-in set, Failure facts (built-in), for every Agent turn. It reads span data that Runtype already stores and makes no model call, so it costs nothing and reads no message content. Its results appear in the signals block, the signal run filter, and stats, like any other set.

A fact records what happened, not whether it mattered. For example, tool_error fires on a timeout that the Agent then retried successfully.

The built-in set is read-only. GET /v1/signal-sets lists it with builtin: true, and every request that would rename, republish, bind, unbind, or preview it returns SIGNAL_SET_BUILTIN. Its facts come from span data with no model call, so a preview would only add cost.

Create a signal set

Start from the conversation_health preset or write your own definition. The preset covers frustration, repeated asks, corrections, escalation requests, out-of-scope asks, promised follow-ups, and unanswered questions.

The following request creates a set from a definition:

curl https://api.runtype.com/v1/signal-sets \
  -H "Authorization: Bearer API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "name": "Support follow-ups",
    "definition": {
      "context": { "priorTurns": 4 },
      "signals": {
        "promised_followup": { "type": "bool", "instructions": "The assistant promised to follow up later." },
        "frustration": { "type": "choice", "options": ["none", "mild", "churn_threat"], "notable": ["churn_threat"] }
      }
    }
  }'

Replace API_KEY with your API key. priorTurns (0 to 20) adds that many earlier turns of the same conversation as context. Version 1 of the set is current as soon as it’s created.

To start from the preset instead, send "preset": "conversation_health" in place of definition. To list presets, call GET /v1/signal-sets/presets.

Preview the set before binding it

A preview runs one version over the Agent’s most recent completed traces and writes nothing. The model calls are billed.

curl https://api.runtype.com/v1/signal-sets/SIGNAL_SET_ID/versions/1/preview \
  -H "Authorization: Bearer API_KEY" \
  -H "Content-Type: application/json" \
  -d '{ "agentId": "AGENT_ID", "limit": 20 }'

Replace SIGNAL_SET_ID with the set’s sigset_ ID and AGENT_ID with the Agent’s ID. limit defaults to 20 and accepts at most 50.

The report lists each signal’s fire rate, mean confidence, review-band count, and sample answers with source spans, plus a projected monthly cost. A signal that fires on most traces usually has instructions that are too broad. To change a signal, publish a new version with POST /v1/signal-sets/SIGNAL_SET_ID/versions and preview that version.

Bind the set to an Agent

A binding starts evaluation on every matching turn from then on.

curl https://api.runtype.com/v1/signal-bindings \
  -H "Authorization: Bearer API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "signalSetId": "SIGNAL_SET_ID",
    "target": { "kind": "agent", "id": "AGENT_ID" },
    "budget": { "monthlyUsd": 10 }
  }'

The binding uses the set’s current version unless you pass versionNumber. It accepts the following optional fields:

Publishing a new version doesn’t move an existing binding. To run a newer version, unbind with DELETE /v1/signal-bindings/BINDING_ID and bind again.

Refusals

Runtype refuses some bindings, and refuses a preview for the same reasons because a preview also sends turns to the model. The default models run on Cloudflare Workers AI, which retains no data and is covered by Runtype’s BAA, so organizations under zero data retention or handling protected health information bind them with no extra step. The last two refusals apply only to a set whose decisionModel is outside Workers AI. Relay these refusals to a person instead of working around them:

Read the results

Once a set is bound, every trace read carries a signals block: GET /v1/runs, GET /v1/runs/EXECUTION_ID, the trace execution and trace conversation reads, and the log list. A run’s block has a status, the keys that fired, and on detail reads the full results with each answer’s value, confidence, band, and source spans.

The status takes one of the following values:

• complete: every turn of the run has results.
• pending: evaluation hasn’t finished. This usually takes a few seconds after the trace completes.
• partial: some turns of the run are still pending.
• unavailable: results couldn’t be read. The rest of the response is unaffected.
• none: no binding covered the run.

To list only the runs where a signal fired, pass signal. Use the key alone, or key:value for one choice value:

curl "https://api.runtype.com/v1/runs?agentId=AGENT_ID&signal=frustration:churn_threat" \
  -H "Authorization: Bearer API_KEY"

In the dashboard, the Activity panel shows the fired keys on each run and adds a signal filter. The run pane lists each answer and quotes its source text. An Agent with a binding shows a read-only Signals section with fire rates by set version.

Read rollup stats

GET /v1/signals/stats returns hourly counters summed over a window (default: the last 7 days). Each row counts evaluated, fired, review-band, skipped, and failed turns, and cost, by binding, set version, signal key, value, and sample.

Pass sample=uniform for rates on live traffic. filtered counts describe only the traffic the binding selects. Don’t add the two together.

Configure signals from other clients

Every operation is available outside the REST API:

• MCP server: list_signal_sets, get_signal_set, create_signal_set, update_signal_set, publish_signal_set_version, preview_signal_set, list_signal_bindings, bind_signal_set, unbind_signal_set, and get_signal_stats. list_runs accepts signal. An MCP client can also read the trace-signals documentation topic, so you can ask a coding agent to “tell me when support conversations go badly” and it configures the set.
• SDK: Runtype.signals with listSets, listPresets, getSet, createSet, updateSet, publishVersion, preview, listBindings, bind, unbind, and stats.
• CLI: runtype signals list, get, create, update, publish, preview, bindings, bind, unbind, and stats, plus runtype runs list --signal KEY.

Error codes

Configuration requests return these codes in code:

Next steps

• Improving your agents from production traces: turn a run where a signal fired into an eval case.
• Runtype MCP server: connect a coding agent that can configure signals for you.
• Runtype CLI reference: the full runtype signals command group.