> For clean Markdown of any page, append .md to the page URL.
> For a complete documentation index, see https://docs.runtype.com/llms.txt.
> For AI client integration (Claude Code, Cursor, etc.), connect to the MCP server at https://docs.runtype.com/_mcp/server.

# Read signals out of every agent turn

> Configure trace signals, preview them on real traces, bind them to an agent, and filter runs by what fired

Trace signals are typed fields that Runtype reads out of every completed turn of an Agent, such as "did the assistant promise a follow-up?" or "how frustrated is the user?". This guide shows you how to create a signal set, preview it on recent traces, bind it to an Agent, and read the results on runs and traces.

You configure signals through the Runtype Model Context Protocol (MCP) server, the REST API, the SDK, or the CLI. The dashboard only displays them.

Every organization also gets the built-in **Failure facts** set without configuring anything. See [Failure facts on every Agent](#failure-facts-on-every-agent).

## Before you begin

You need the following:

* A Runtype Agent, native or external, with completed executions in the last 30 days. An external Agent sends its traces to Runtype over OpenTelemetry. For details, see [Reporting telemetry from an external agent](/developer-guides/guides/reporting-external-telemetry).
* For the REST API, SDK, or CLI, an API key with the `SIGNALS:WRITE` permission. Reading results and stats needs `SIGNALS:READ`. For details, see [Authentication](/developer-guides/getting-started/authentication).

## How signals are evaluated

A signal set lists up to 40 signals. Each signal has a key, a type, and instructions. Runtype sends each turn's signals to two kinds of model in one call per kind:

| Signal type                                              | Evaluated by        | Answer                                                |
| -------------------------------------------------------- | ------------------- | ----------------------------------------------------- |
| `bool`, `choice`                                         | A decision model    | A value with a calibrated confidence                  |
| `text`, `number`, `date`, `duration`, `list`, `estimate` | An extraction model | A value, or `null`, with confidence and a source span |

Each answer falls into a band. An answer at or above `bands.act` (default 0.9) is in the act band. An answer between `bands.review` (default 0.6) and `bands.act` is in the review band: count it, but don't act on it without a person checking it.

A signal **fires** when a `bool` is `true`, or a `choice` value is listed in `notable`, in the act band. Extraction signals carry values and never fire on their own.

## Failure facts on every Agent

Runtype computes a built-in set, `Failure facts (built-in)`, for every Agent turn. It reads span data that Runtype already stores and makes no model call, so it costs nothing and reads no message content. Its results appear in the `signals` block, the `signal` run filter, and stats, like any other set.

| Key                    | Fires when                                                                                  |
| ---------------------- | ------------------------------------------------------------------------------------------- |
| `run_failed`           | The execution ended with status `error`.                                                    |
| `tool_error`           | A tool call failed. The value is the error class, such as `timeout` or `rate_limited`.      |
| `tool_repeated`        | One tool ran 3 or more times in the turn with exactly the same arguments.                   |
| `invalid_tool_input`   | The model sent tool input that failed validation, or Runtype repaired a tool input or name. |
| `turn_limit_exhausted` | The agent loop stopped on its turn limit.                                                   |
| `abrupt_end`           | The last model call stopped on `length` or `content_filter`, or reported no finish reason.  |
| `latency_outlier`      | The turn took more than 3 times the Agent's 7-day p95 duration. Needs 20 turns of history.  |
| `cost_outlier`         | The turn cost more than 3 times the Agent's 7-day p95 cost. Needs 20 turns of history.      |

A fact records what happened, not whether it mattered. For example, `tool_error` fires on a timeout that the Agent then retried successfully.

The built-in set is read-only. `GET /v1/signal-sets` lists it with `builtin: true`, and every request that would rename, republish, bind, unbind, or preview it returns `SIGNAL_SET_BUILTIN`. Its facts come from span data with no model call, so a preview would only add cost.

## Create a signal set

Start from the `conversation_health` preset or write your own definition. The preset covers frustration, repeated asks, corrections, escalation requests, out-of-scope asks, promised follow-ups, and unanswered questions.

The following request creates a set from a definition:

**`cURL`**

```bash title="cURL"
curl https://api.runtype.com/v1/signal-sets \
  -H "Authorization: Bearer API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "name": "Support follow-ups",
    "definition": {
      "context": { "priorTurns": 4 },
      "signals": {
        "promised_followup": { "type": "bool", "instructions": "The assistant promised to follow up later." },
        "frustration": { "type": "choice", "options": ["none", "mild", "churn_threat"], "notable": ["churn_threat"] }
      }
    }
  }'
```

Replace `API_KEY` with your API key. `priorTurns` (0 to 20) adds that many earlier turns of the same conversation as context. Version 1 of the set is current as soon as it's created.

To start from the preset instead, send `"preset": "conversation_health"` in place of `definition`. To list presets, call `GET /v1/signal-sets/presets`.

## Preview the set before binding it

A preview runs one version over the Agent's most recent completed traces and writes nothing. The model calls are billed.

**`cURL`**

```bash title="cURL"
curl https://api.runtype.com/v1/signal-sets/SIGNAL_SET_ID/versions/1/preview \
  -H "Authorization: Bearer API_KEY" \
  -H "Content-Type: application/json" \
  -d '{ "agentId": "AGENT_ID", "limit": 20 }'
```

Replace `SIGNAL_SET_ID` with the set's `sigset_` ID and `AGENT_ID` with the Agent's ID. `limit` defaults to 20 and accepts at most 50.

The report lists each signal's fire rate, mean confidence, review-band count, and sample answers with source spans, plus a projected monthly cost. A signal that fires on most traces usually has instructions that are too broad. To change a signal, publish a new version with `POST /v1/signal-sets/SIGNAL_SET_ID/versions` and preview that version.

## Bind the set to an Agent

A binding starts evaluation on every matching turn from then on.

**`cURL`**

```bash title="cURL"
curl https://api.runtype.com/v1/signal-bindings \
  -H "Authorization: Bearer API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "signalSetId": "SIGNAL_SET_ID",
    "target": { "kind": "agent", "id": "AGENT_ID" },
    "budget": { "monthlyUsd": 10 }
  }'
```

The binding uses the set's current version unless you pass `versionNumber`. It accepts the following optional fields:

| Field          | Default                | Description                                                                                     |
| -------------- | ---------------------- | ----------------------------------------------------------------------------------------------- |
| `coverage`     | `{ "mode": "all" }`    | `{ "mode": "sample", "rate": 0.2 }` evaluates that share of matching turns.                     |
| `filter`       | `{}`                   | `status` limits evaluation to traces with those statuses. `minTurns` skips short conversations. |
| `uniformShare` | `0.05`                 | A share of turns drawn at random before the filter, so live-traffic rates stay unbiased.        |
| `budget`       | `{ "monthlyUsd": 25 }` | When the month's budget is spent, Runtype skips evaluation and counts each skipped turn.        |

Publishing a new version doesn't move an existing binding. To run a newer version, unbind with `DELETE /v1/signal-bindings/BINDING_ID` and bind again.

### Refusals

Runtype refuses some bindings, and refuses a preview for the same reasons because a preview also sends turns to the model. The default models run on Cloudflare Workers AI, which retains no data and is covered by Runtype's BAA, so organizations under zero data retention or handling protected health information bind them with no extra step. The last two refusals apply only to a set whose `decisionModel` is outside Workers AI. Relay these refusals to a person instead of working around them:

| Code                                         | Status | Meaning                                                                                                                                                                                                                                                                 |
| -------------------------------------------- | ------ | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `SIGNALS_LOGGING_OFF`                        | 422    | The Agent has `loggingPolicy: "off"`, so there are no turns to read.                                                                                                                                                                                                    |
| `SIGNALS_ZERO_DATA_RETENTION`                | 422    | The organization is under zero data retention, and the set's decision model is outside Workers AI. Remove `decisionModel` to use the default.                                                                                                                           |
| `SIGNALS_EXTERNAL_PROCESSING_UNACKNOWLEDGED` | 409    | The organization handles protected health information, and the set's decision model is outside Workers AI. Remove `decisionModel`, or retry with `"acknowledgeExternalProcessing": true` after a person confirms. Runtype records the acknowledgement in the audit log. |

## Read the results

Once a set is bound, every trace read carries a `signals` block: `GET /v1/runs`, `GET /v1/runs/EXECUTION_ID`, the trace execution and trace conversation reads, and the log list. A run's block has a `status`, the keys that `fired`, and on detail reads the full `results` with each answer's value, confidence, band, and source spans.

The `status` takes one of the following values:

* **`complete`**: every turn of the run has results.
* **`pending`**: evaluation hasn't finished. This usually takes a few seconds after the trace completes.
* **`partial`**: some turns of the run are still pending.
* **`unavailable`**: results couldn't be read. The rest of the response is unaffected.
* **`none`**: no binding covered the run.

To list only the runs where a signal fired, pass `signal`. Use the key alone, or `key:value` for one `choice` value:

**`cURL`**

```bash title="cURL"
curl "https://api.runtype.com/v1/runs?agentId=AGENT_ID&signal=frustration:churn_threat" \
  -H "Authorization: Bearer API_KEY"
```

In the dashboard, the **Activity** panel shows the fired keys on each run and adds a signal filter. The run pane lists each answer and quotes its source text. An Agent with a binding shows a read-only **Signals** section with fire rates by set version.

## Read rollup stats

`GET /v1/signals/stats` returns hourly counters summed over a window (default: the last 7 days). Each row counts evaluated, fired, review-band, skipped, and failed turns, and cost, by binding, set version, signal key, value, and sample.

Pass `sample=uniform` for rates on live traffic. `filtered` counts describe only the traffic the binding selects. Don't add the two together.

## Configure signals from other clients

Every operation is available outside the REST API:

* **MCP server**: `list_signal_sets`, `get_signal_set`, `create_signal_set`, `update_signal_set`, `publish_signal_set_version`, `preview_signal_set`, `list_signal_bindings`, `bind_signal_set`, `unbind_signal_set`, and `get_signal_stats`. `list_runs` accepts `signal`. An MCP client can also read the `trace-signals` documentation topic, so you can ask a coding agent to "tell me when support conversations go badly" and it configures the set.
* **SDK**: `Runtype.signals` with `listSets`, `listPresets`, `getSet`, `createSet`, `updateSet`, `publishVersion`, `preview`, `listBindings`, `bind`, `unbind`, and `stats`.
* **CLI**: `runtype signals list`, `get`, `create`, `update`, `publish`, `preview`, `bindings`, `bind`, `unbind`, and `stats`, plus `runtype runs list --signal KEY`.

## Error codes

Configuration requests return these codes in `code`:

| Code                                                      | Fix                                                                                      |
| --------------------------------------------------------- | ---------------------------------------------------------------------------------------- |
| `SIGNAL_SET_INVALID`                                      | `details.issues` lists one sentence and `path` per problem. Fix each field and resubmit. |
| `SIGNAL_SET_DEFINITION_REQUIRED`                          | Pass `preset` or `definition`.                                                           |
| `SIGNAL_SET_PRESET_UNKNOWN`                               | Use a preset ID from `GET /v1/signal-sets/presets`.                                      |
| `SIGNAL_SET_BUILTIN`                                      | The built-in set is read-only. Create your own set to change what is evaluated.          |
| `SIGNAL_SET_NAME_TAKEN`                                   | Choose another name, or publish a new version of the existing set.                       |
| `SIGNAL_SET_NOT_FOUND` and `SIGNAL_SET_VERSION_NOT_FOUND` | Use a set ID and version number from `GET /v1/signal-sets`.                              |
| `SIGNAL_TARGET_NOT_FOUND`                                 | Use an Agent ID from your organization.                                                  |
| `SIGNAL_BINDING_INVALID`                                  | Fix the coverage rate, uniform share, or budget that the message names.                  |
| `SIGNAL_BINDING_NOT_FOUND`                                | Use a binding ID from `GET /v1/signal-bindings`.                                         |
| `SIGNAL_STATS_WINDOW_INVALID`                             | Pass `from` and `to` as ISO 8601 timestamps.                                             |

## Next steps

* [Improving your agents from production traces](/developer-guides/guides/improving-agents-from-production): turn a run where a signal fired into an eval case.
* [Runtype MCP server](/developer-guides/integrations/runtype-mcp-server): connect a coding agent that can configure signals for you.
* [Runtype CLI reference](/developer-guides/getting-started/cli-reference): the full `runtype signals` command group.