> For clean Markdown of any page, append .md to the page URL. > For a complete documentation index, see https://docs.runtype.com/llms.txt. > For AI client integration (Claude Code, Cursor, etc.), connect to the MCP server at https://docs.runtype.com/_mcp/server. # Instrumenting a Flue agent > Send Flue agent runs to Runtype with @runtypelabs/flue-otel `@runtypelabs/flue-otel` is OpenTelemetry instrumentation from Runtype for [Flue](https://flueframework.com) agents. Install it with Flue's `instrument()` function, configure an OTLP exporter for Runtype, and view each instrumented run in the dashboard. Runtype displays the model, token usage, cost, loop iterations, tool calls, and stop reason. Your agent continues to run with your model credentials. Runtype provides the dashboard where you inspect the run. ## Choose a path Flue runs on several hosts, and the shortest route to a run in the dashboard depends on where your agent runs and what you need from it. | Your agent | Use | | ----------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Deployed to Cloudflare Workers | No code. Export Workers traces to Runtype: [Deploy on Cloudflare Workers](#deploy-on-cloudflare-workers). | | On Node | `@runtypelabs/flue-otel`, this guide. It sends the transcript, system prompt and tool content by default (each kind can be switched off), so runs can be captured as eval cases, and it reports an exact iteration count, a run-level stop reason, and summed usage. | | On Node, stock instrumentation only | `@flue/opentelemetry`. It exports the conversation by default, so runs carry a transcript and can be captured as eval cases. Runtype derives loop structure from Flue's own attributes; the exact iteration count and run-level stop reason are unavailable. | | Something Runtype calls | Register the agent as an `external` agent with a `runtype-stream` endpoint: [Let Runtype call your Flue agent](#let-runtype-call-your-flue-agent). Combine with either telemetry path. | For the full picture of which Runtype features apply to an agent that runs elsewhere, read [Bring your own agent](/developer-guides/guides/bring-your-own-agent). ## When to use this instead of a generic exporter [Reporting telemetry from an external agent](/developer-guides/guides/reporting-external-telemetry) covers the general path. A generic exporter sends the GenAI semantic-convention attributes that your spans contain. This package adds two values that a generic export cannot derive after the spans leave the process: * **An exact loop iteration count.** Flue events do not include an absolute ordinal. If you rank turn IDs after export, you count only the turns in the export batch. A five-turn run whose final batch has two turns appears to have two iterations. The instrumentation counts each agent turn before export. * **A run-level stop reason and summed token usage.** The instrumentation sums token counts from model turns and adds the total to the run's `invoke_agent` span. The run span retains that total when model spans flush in separate batches. ## Before you begin You need the following: * Node 22 or later. * A Flue runtime from `1.0.0-beta.9` onward, or Flue 2.x. The package supports both through one entry point. * A Runtype agent that receives the runs, and its agent ID. * A telemetry API key with the **Telemetry Ingest** permission group (`TELEMETRY:WRITE`). Keys with `AGENTS:WRITE` also work. The quickest way to get the agent and the key is the **Connect an existing agent** page in the dashboard. Open it from the command palette and choose **Flue**. Select or create the **Receiving agent**, then click **Create telemetry key**. In an empty workspace, the page creates both for you. The page shows a ready-to-run Flue instrumentation file. When the first run arrives, the page shows **Trace received**. To set them up yourself, create an agent, then follow [Create a telemetry API key](/developer-guides/guides/reporting-external-telemetry#create-a-telemetry-api-key). ## Install and start Install the package and the OpenTelemetry API with this command: **`Install`** ```bash title="Install" npm install @runtypelabs/flue-otel @opentelemetry/api ``` Register the instrumentation with Flue using this code: **`Register the instrumentation`** ```typescript title="Register the instrumentation" import { instrument } from '@flue/runtime' import { createRuntypeFlueInstrumentation } from '@runtypelabs/flue-otel' const stopInstrumenting = instrument(createRuntypeFlueInstrumentation()) ``` `instrument()` returns a disposer, here `stopInstrumenting`. Call it when you stop instrumentation. It ends open spans as `interrupted` and does not flush them, because your application owns the exporter. The package depends on `@opentelemetry/api` and does not include an SDK, provider, exporter, sampler, or resource. Your application configures those components, and the instrumentation writes spans through the provider that you register. If your application already configures OpenTelemetry, register the instrumentation and add the resource attributes that identify the Runtype agent. Use `runtypeFlueResourceAttributes({ agentId })` as described in [Attribution](#attribution-which-runtype-agent-owns-the-run). `createRuntypeFlueInstrumentation()` accepts the following options: | Option | Purpose | | --------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `agents` | Maps each Flue agent name to a Runtype agent ID. See [Run several agents in one process](#run-several-agents-in-one-process). | | `tracer` | The OpenTelemetry tracer that receives the spans. Defaults to the tracer of the globally registered provider. Pass one to send Runtype's spans through a provider of their own. | | `content` | Which content the spans carry. All kinds are on by default. Pass `false` to send no content. See [Control what content is sent](#control-what-content-is-sent). | ## Set up an OpenTelemetry pipeline If your application does not have an OpenTelemetry pipeline, install `@opentelemetry/sdk-trace-node`, `@opentelemetry/exporter-trace-otlp-http`, and `@opentelemetry/resources`. Configure the provider, exporter, and instrumentation with this example: **`Full pipeline`** ```typescript title="Full pipeline" import { NodeTracerProvider, BatchSpanProcessor } from '@opentelemetry/sdk-trace-node' import { OTLPTraceExporter } from '@opentelemetry/exporter-trace-otlp-http' import { resourceFromAttributes } from '@opentelemetry/resources' import { instrument } from '@flue/runtime' import { createRuntypeFlueInstrumentation, runtypeFlueResourceAttributes, } from '@runtypelabs/flue-otel' const provider = new NodeTracerProvider({ resource: resourceFromAttributes({ 'service.name': 'my-agent', ...runtypeFlueResourceAttributes({ agentId: process.env.RUNTYPE_AGENT_ID }), }), spanProcessors: [ new BatchSpanProcessor( new OTLPTraceExporter({ url: 'https://api.runtype.com/v1/otel/v1/traces', headers: { Authorization: `Bearer ${process.env.RUNTYPE_API_KEY}` }, }), { maxExportBatchSize: 32 } ), ], }) provider.register() const stopInstrumenting = instrument(createRuntypeFlueInstrumentation()) process.once('beforeExit', () => void provider.shutdown().catch(console.error)) ``` Set `RUNTYPE_AGENT_ID` to the ID of the Runtype agent that receives the run. Set `RUNTYPE_API_KEY` to the telemetry API key. `maxExportBatchSize` keeps each export request small while content is on. For the reason, see [Size limits](#size-limits). The following settings keep spans in one trace and preserve the closing run span: > **Warning** > > **Register the provider.** `provider.register()` installs the OpenTelemetry > context manager. Without it, the API uses a no-op context manager, so each > span becomes a trace root and Runtype stores separate traces instead of one > run. > > **Flush before the process exits.** A `BatchSpanProcessor` exports child spans > before parent spans. If the process exits without `forceFlush()` or > `shutdown()`, the closing `invoke_agent` span can be lost. A run whose > `invoke_agent` span never arrives remains in progress. In a serverless handler, > await `provider.forceFlush()` before returning. ## Verify the run To confirm that runs reach Runtype, follow these steps: 1. Run your agent once and let the process exit or flush. 2. In the dashboard, open the agent and select the **Runs** tab. 3. Open the new run and check the iteration count, the stop reason, and the transcript. If the run stays in progress, the closing span did not arrive. Check that your application calls `forceFlush()` or `shutdown()` on the provider. If no run appears, see [Troubleshooting](#troubleshooting). ## Attribution: which Runtype agent owns the run Runtype resolves the agent for each trace in this order: 1. The `runtype.agent.id` resource attribute. The `runtypeFlueResourceAttributes({ agentId })` helper sets it. 2. The `x-runtype-agent-id` request header that your exporter sends. 3. The `runtype.agent.id` attribute on the run's `invoke_agent` span. The `agents` option sets it. Use the resource attribute or request header when one process runs one Runtype agent. ### Run several agents in one process When one process runs several agents, map each Flue agent name to a Runtype agent ID with the `agents` option: **`Per agent attribution`** ```typescript title="Per agent attribution" createRuntypeFlueInstrumentation({ agents: { triage: 'YOUR_TRIAGE_AGENT_ID', billing: 'YOUR_BILLING_AGENT_ID', }, }) ``` Replace `YOUR_TRIAGE_AGENT_ID` and `YOUR_BILLING_AGENT_ID` with the corresponding Runtype agent IDs. The package reports a delegated sub-agent as a tool call on the parent run. The sub-agent's work stays in the delegating agent's trace, and one trace represents one execution, so delegation does not create a second run. > **Warning** > > **Keep one agent invocation per trace.** Spans inherit the active OpenTelemetry > context. If an enclosing span remains active when two agent invocations run, > both invocations land in the same trace. An HTTP server span from > auto-instrumentation commonly causes this. > > Runtype either merges the invocations into one execution, which drops the > second run's usage and stop reason, or rejects the trace as > `ambiguous_agent_attribution` when the invocations claim different IDs through > `agents` and no resource attribute or request header resolves the trace. The > rejected trace produces no run. > > To dispatch several agent invocations in one request or job, start each > invocation in its own trace. A separate process per agent lets you set the > resource attribute instead. Flue's `dispatch()` does not propagate trace > context, so a direct `dispatch()` call creates its own trace. ## What it emits The instrumentation emits the following span types and attributes: | Span | When | Attributes | | ------------------------------------------------------ | -------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `invoke_agent` | One per agent invocation | The request and response model, summed token usage, `runtype.stop_reason`, `runtype.tools.reported`, the highest `runtype.iteration`, `runtype.execution.id`, `runtype.adapter.name` and `runtype.adapter.version`, and `runtype.agent.id` when the `agents` option maps the agent. | | `chat` | One per model turn | The provider, request and response model, finish reason, per-turn usage, `runtype.turn.id`, `runtype.turn.index`, `runtype.iteration`, and `runtype.provider.finish_reason` / `runtype.gateway.log_id` when the provider records them. | | `execute_tool` | One per model-requested tool call | `gen_ai.tool.name`, `gen_ai.tool.call.id`, the loop position, and `runtype.tool.type` when the package identifies the tool class. | | `flue.task`, `flue.compaction`, `flue.operation shell` | Delegation, compaction, and host shell calls | Correlation IDs only. | The package sets `runtype.stop_reason` to one of these values: `end_turn`, `length`, `content_filter`, `error`, or `unknown`. Every span also carries the `flue.*` correlation attributes that the standard `@flue/opentelemetry` instrumentation emits. These attributes support dashboards that group spans by Flue correlation keys. The following details explain two intentional omissions: #### Intentional omissions **`runtype.tool.type` for ordinary tools.** Flue's `origin` field identifies who initiated a call, not its implementation class. The package emits this attribute only for a sub-agent delegation and maps a 1.x `datastore` tool to `data_connection`. **An unknown stop reason.** If a run ends during a tool call, the package reports `unknown`. It cannot identify whether a turn cap, tool cap, or host abort stopped the run. ## Compose with other instrumentations Flue's `instrument()` supports multiple subscribers. This package uses a separate instrumentation key, so it does not replace `@flue/opentelemetry`. > **Warning** > > **Send one Flue instrumentation to each Runtype endpoint.** If this package and the standard > `@flue/opentelemetry` instrumentation export to the same endpoint, Runtype receives two > `invoke_agent` spans for one run and counts the token usage and cost twice. You can send them to > different backends. If you already export Flue traces to Runtype with the standard `@flue/opentelemetry` instrumentation, replace it with this package for that endpoint. If your application already exports to another backend, pass a `tracer` from a separate provider that exports only to Runtype. Runtype's spans then leave through that provider, and the rest of the process keeps its own exporter. ## Control what content is sent By default, the package sends the conversation, the final output, the system prompt, and each tool's arguments and result. Each kind has its own switch, and every switch defaults to on. Runs need the conversation to show a transcript and to be captured as eval cases. To send shape and cost only, with no content of any kind, pass `content: false`: **`No content`** ```ts title="No content" instrument(createRuntypeFlueInstrumentation({ content: false })) ``` ### Message content Three switches control the content on the run's `invoke_agent` span: **`Message content switches`** ```ts title="Message content switches" instrument( createRuntypeFlueInstrumentation({ content: { inputMessages: true, outputMessages: true, systemInstructions: false, }, }) ) ``` * `inputMessages` is the conversation as the run's last model turn saw it, including user messages sent while the run was in progress. Runtype shows it as the run's transcript and uses it when you capture the run as an eval case. * `outputMessages` is the final assistant text of the run. Runtype shows it as the run's final output. * `systemInstructions` is the system prompt. It is usually the largest value a run carries and rarely differs between runs, so it is the switch most deployments turn off. Messages carry text only. Tool calls, tool results, thinking and image parts, and `system` rows in the history are left out. Message content behaves the same on Flue 1.x and 2.x. ### Tool content Each `execute_tool` span carries the tool's arguments and result, shown on the tool's card in the trace view. To drop either, switch it off: **`Tool content switches`** ```ts title="Tool content switches" instrument( createRuntypeFlueInstrumentation({ content: { toolArguments: true, toolResults: false, }, }) ) ``` Runtype shows arguments as a JSON object. The package wraps a non-object value as `{ "value": ... }`. A string result is sent as-is, and any other result is sent as JSON. A delegation to a sub-agent is a tool call whose arguments carry the prompt handed to that sub-agent. To keep that prompt out of the export, return `undefined` from `redact` for the delegation tool, as shown in [Redact content](#redact-content). ### Size limits `maxChars` sets the longest value the package sends, including the truncation marker. It defaults to 64 KiB and is capped at 256 KiB, because Runtype drops a single content attribute larger than that whole instead of cutting it. Runtype accepts at most 384 KiB of content on one span. To stay under that limit, the package applies lower caps whatever `maxChars` says: * Tool arguments and tool results are each capped at 192 KiB. * Each message value and the system prompt on the run span are capped at 128 KiB. When a value is too long, the package shortens it so that it still reads as incomplete: * A result is cut and ends with `…[truncated N chars]`. * Arguments are replaced by `{ "truncated": "..." }`, which holds the start of the JSON and the same marker, so they stay a valid object. * A message list drops its oldest messages first, so the newest survive. Runtype also limits the total content of one trace, and drops content past that limit. For the limit, see [Capture conversation content](/developer-guides/guides/reporting-external-telemetry#capture-conversation-content). Content makes spans large. One export request can carry at most 8 MiB, and a larger request is rejected whole. A batch of 512 spans, the `BatchSpanProcessor` default, can exceed that. Set `maxExportBatchSize` to 16 to 32 for agents with many tool calls, or 64 for short runs. ### Redact content `redact` runs on every content value before encoding and truncation. It receives the value and a context whose `kind` names the content: | `kind` | Value it receives | What to return | | ----------------------------------- | ------------------------------------------------------------------ | ------------------------------------------------- | | `arguments`, `result` | The raw tool payload. `context.toolName` names the tool. | The value, or a replacement. | | `input_messages`, `output_messages` | An array of `{ role, parts: [{ type: 'text', content }] }` objects | The array, or a replacement array of that shape. | | `system_instructions` | The system prompt as a string | A string or text parts. Anything else sends none. | Return `undefined` to drop that one attribute. If `redact` throws, the package drops that attribute and sends the rest of the span. The following example masks email addresses in the conversation and drops the prompt that a `delegate` tool hands to a sub-agent: **`Redact content`** ```ts title="Redact content" import { instrument } from '@flue/runtime' import { createRuntypeFlueInstrumentation, type FlueContentMessage, } from '@runtypelabs/flue-otel' const EMAIL = /[^\s@]+@[^\s@]+\.[^\s@]+/g instrument( createRuntypeFlueInstrumentation({ content: { redact: (value, context) => { if (context.kind === 'arguments' && context.toolName === 'delegate') return undefined if (context.kind !== 'input_messages') return value return (value as FlueContentMessage[]).map((message) => ({ ...message, parts: message.parts.map((part) => ({ ...part, content: part.content.replace(EMAIL, '[email]'), })), })) }, }, }) ) ``` ### What it does not send Whatever you set, the package never sends error messages or stack traces, a failed tool's result, content from the host's own `session.shell()` call, thinking or image parts, or Flue's internal signal messages. With `content: false`, the package sends only identifiers, structure, metrics, model IDs, token counts, durations, tool names, correlation IDs, error types, and exception class names for failed spans. This has two consequences: * **Runs have no transcript.** You can inspect timing, cost, iteration counts, and the tool call sequence, but not the exchanged text. * **Eval capture is unavailable.** Capturing an external run into an eval suite requires the transcript. > **Note** > > The standard `@flue/opentelemetry` instrumentation exports content by default. Review its exporter > configuration before sending traces to a third-party backend. ## Deploy on Cloudflare Workers Workers Observability traces a Flue 2 agent deployed with `@flue/vite` and `@cloudflare/vite-plugin` automatically. It emits an `invoke_agent` span per run, a `chat` span per model turn, and an `execute_tool` span per tool call, in the same GenAI semantic conventions that this package emits, with conversation content included by default. You do not install this package or an OpenTelemetry SDK. Export the Workers traces to Runtype instead. To export Workers traces to Runtype, follow these steps: 1. In the Cloudflare dashboard, open **Workers Observability** and add a destination of type **Traces** with the endpoint `https://api.runtype.com/v1/otel/v1/traces` and two custom headers: `Authorization: Bearer rt_YOUR_TELEMETRY_KEY` and `x-runtype-agent-id: agent_YOUR_AGENT_ID`. Replace `rt_YOUR_TELEMETRY_KEY` with your telemetry API key and `agent_YOUR_AGENT_ID` with the ID of the Runtype agent that receives the runs. Exporting Workers traces to an OTLP destination requires a Workers Paid plan. 2. Reference the destination in `wrangler.jsonc`: **`wrangler.jsonc`** ```jsonc title="wrangler.jsonc" { "observability": { "traces": { "enabled": true, "destinations": ["runtype"], "head_sampling_rate": 1, }, }, } ``` 3. Run `npx vite build && npx wrangler deploy` and start a conversation. Each successful response appears under the agent's **Runs** with a transcript, so you can capture it as an eval case. Each response produces its own trace, and Runtype records each trace as a separate run, even when two messages share a conversation. A canceled response might not produce a completed run. To remove the content, register `instrument(createCloudflareTracing({ content: false }))` from `@flue/runtime/cloudflare` at module scope in `app.ts`. The runs then lose eval capture as well. ## Let Runtype call your Flue agent Telemetry sends data from your agent to Runtype. To let Runtype call your agent, register it as an `external` agent whose endpoint streams Runtype execution events. You can then test it from the dashboard, put a Persona chat widget or a surface in front of it, or run it on a schedule. Create the agent with this request: **`POST /v1/agents`** ```json title="POST /v1/agents" { "name": "Support agent (Flue)", "agentType": "external", "externalConfig": { "endpoint": "https://my-worker.example.workers.dev/dispatch", "protocol": "runtype-stream", "framework": "flue", "auth": { "type": "bearer", "credentials": "{{secret:MY_AGENT_TOKEN}}" } } } ``` The endpoint receives `POST {endpoint}` with `{ "messages": [...], "context": { "conversationId": "..." } }`. It answers with a `text/event-stream` of execution events: `execution_start` first, then one terminal `execution_complete` or `execution_error` last. The `ExecutionStreamEvent` union in the [API reference](/api-reference) lists every event. In a Flue 2 app, the route runs the turn with `init(agent, { id: conversationId })` and `handle.dispatch(message)`, then maps each chunk from `handle.read(receipt, { onEvent })` onto the matching event. Eval suites do not run against a `runtype-stream` endpoint. They run against an A2A endpoint. For the feature matrix, see [Bring your own agent](/developer-guides/guides/bring-your-own-agent). ## Troubleshooting The following table lists common export problems: | Symptom | Cause | Fix | | ------------------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------ | | The exporter reports `403` | The request has no Runtype API key, or the key has neither `TELEMETRY:WRITE` nor `AGENTS:WRITE`. | Send a telemetry API key in the `Authorization` header. | | The exporter reports `400` | No agent ID resolves the trace, or its invocations claim different agents. | Set the resource attribute or the `x-runtype-agent-id` header. See [Attribution](#attribution-which-runtype-agent-owns-the-run). | | The exporter reports `404` | The key cannot reach the agent ID that the trace names. | Check the agent ID, and use a key from the organization that owns the agent. | | The exporter reports `413` | The export request or trace is too large. | Lower `maxExportBatchSize` or `maxChars`. If one trace is too large, keep one agent invocation per trace. See [Size limits](#size-limits). | | Some runs are missing, with no error | Runtype rejected some traces in the export. It answers `200` with a `partial_success` body that names each rejected trace, and many exporters do not log that body. | Log the response body, or check ingest health. | | A run stays in progress | The closing `invoke_agent` span never arrived. | Call `forceFlush()` or `shutdown()` on the provider before the process exits. | To check what Runtype received for a trace, use [Inspect ingest health](/developer-guides/guides/reporting-external-telemetry#inspect-ingest-health). ## Next steps Continue with these guides: * [Bring your own agent](/developer-guides/guides/bring-your-own-agent): which Runtype features apply to an agent that runs elsewhere, and how to wire each. * [Improving your agents from production traces](/developer-guides/guides/improving-agents-from-production): turn ingested runs into evals and regression cases. * [Authentication](/developer-guides/getting-started/authentication): review API key scopes and authentication options.