Add voice to an external agent | Runtype Documentation

An agent that runs on your own infrastructure can answer browser voice calls through Runtype. Register it as an external agent, turn on voice, and start calls with the Runtype browser voice client.

Runtype handles the voice side of the call: microphone audio, speech-to-text, turn-taking and interruptions, text-to-speech, and the saved conversation. Your agent only receives text and returns text. It never handles audio.

How a voice turn reaches your agent

When the caller finishes speaking, Runtype transcribes the speech and checks that the caller owns the conversation. It then calls your agent with the new transcript and appends both the user message and your agent’s reply to the saved conversation.

What your agent receives depends on its protocol. The following table compares the two protocols:

Protocol	What your agent receives	When the caller hears the reply
`runtype-stream`	The full conversation history in `messages`, and the saved conversation’s ID in `context.conversationId`, on every turn.	As your agent streams text.
`a2a`	Only the most recent user message, with the same `contextId` on every turn of the conversation.	As chunks arrive when the agent card advertises `capabilities.streaming: true`. Otherwise, after the reply completes.

An A2A agent receives no earlier turns, so it must keep its own history under the contextId. Runtype reads capabilities.streaming from the agent card that it cached when you registered the agent. After you turn on streaming in your agent card, send POST /v1/agents/{id}/refresh-agent-card to update the cached copy.

For a runtype-stream agent, Runtype speaks only text from your agent’s own execution. Text inside a tool call isn’t spoken. If your execution_complete frame sets finalOutput, Runtype uses it as the reply text in the transcript and the saved conversation. If your agent streams no text, Runtype speaks finalOutput.

Before you begin

Prepare the following:

An agent endpoint that Runtype can call over public HTTPS, using the runtype-stream protocol or A2A. For the endpoint rules and the externalConfig fields, see Bring your own agent.
A Runtype API key that can create and update agents.
A client token that can access the agent, for the browser. For more information, see Client tokens and domain restrictions.
If your endpoint requires authentication, a managed secret that holds its credential. For more information, see Managing secrets.

Register the agent with voice turned on

Send POST /v1/agents with agentType: "external", your externalConfig, and a config.voice block. The following request registers a runtype-stream agent that uses Cloudflare speech:

cURL

curl -X POST https://api.runtype.com/v1/agents \
  -H "Authorization: Bearer YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "name": "Support agent (voice)",
    "agentType": "external",
    "externalConfig": {
      "endpoint": "https://agents.example.com/dispatch",
      "protocol": "runtype-stream",
      "auth": {
        "type": "bearer",
        "credentials": "{{secret:SUPPORT_AGENT_TOKEN}}"
      },
      "timeoutMs": 60000
    },
    "config": {
      "voice": {
        "enabled": true,
        "interruptionMode": "barge-in",
        "speech": {
          "mode": "pipeline",
          "transcriber": { "provider": "cloudflare" },
          "synthesizer": { "provider": "cloudflare" }
        }
      }
    }
  }'

Replace the following:

YOUR_API_KEY: your Runtype API key.
https://agents.example.com/dispatch: your agent’s endpoint.
SUPPORT_AGENT_TOKEN: the name of the managed secret that holds your agent’s bearer token.

The MCP create_agent tool takes the same values as agent_type, external_config, and config.voice. To turn on voice for an agent that you already registered, send PUT /v1/agents/{id} or use the MCP update_agent tool with the same config.voice block. An update replaces the whole voice block, so send every field that you want to keep.

The voice block takes the following settings:

enabled: Set to true to accept voice calls. A call to an agent without it returns HTTP 403 with Agent does not support voice.
interruptionMode: none (the caller can’t interrupt a reply), cancel (the caller stops a reply from the client), or barge-in (the caller interrupts by speaking). It defaults to barge-in for Cloudflare speech and none for ElevenLabs speech.
speech: Set mode to pipeline and use the same provider, cloudflare or elevenlabs, for transcriber and synthesizer. Without a speech block, the agent uses ElevenLabs speech.

Runtype resolves each {{secret:NAME}} reference on the server, in the scope of the account that owns the agent, at the time of each call. The browser and the client token never carry the credential. If a resolved credential appears in your agent’s reply or error, Runtype removes it before it speaks, saves, or logs the text.

Start a voice call

Start calls with the @runtypelabs/voice client. For the install command, the React hook, and the call controls, see Build a custom voice interface.

An external agent answers only on a call that saves its conversation:

Cloudflare speech: Every call saves its conversation. You don’t need extra options.
ElevenLabs speech: A call saves its conversation only when the client passes a visitorToken or a sessionId. Without one, your agent isn’t called, and the caller hears Voice mode does not yet support external agents.

The following hook passes a visitor token for an ElevenLabs call:

React

import { useVoiceClient } from '@runtypelabs/voice/react'
const voice = useVoiceClient({
  agentId,
  clientToken,
  visitorToken: () => visitorStore.get(clientToken),
})

The visitor token is the visitor.token value that POST /v1/client/init returns. For how to request and store it, see Conversation history in an embedded widget.

Keep replies responsive

The caller hears nothing until your agent returns text. We recommend that you use a runtype-stream endpoint, or an A2A agent whose card advertises streaming, so speech starts with the first sentence. An A2A agent without streaming makes the caller wait for the whole reply.

Write replies to be spoken. Keep them short, and leave out Markdown, tables, and long lists.

Interruptions, timeouts, and failures

When the caller interrupts a reply, cancels it, or hangs up, Runtype cancels the request to your agent. It discards any output that arrives later, so that output never reaches the next turn. The saved conversation keeps the part of the reply that streamed before the interruption. When the voice client reports playback progress, Runtype trims the saved reply to the part the caller heard. Runtype records the run as cancelled.

timeoutMs (default 30000) and retryCount (default 3) apply as they do for other calls to your agent. The timeout covers the whole reply, so raise it for turns that take longer than 30 seconds. Runtype retries a request only when it never reached your agent, and it never replays text the caller already heard.

If your agent fails or doesn’t finish within timeoutMs, the voice client receives the error Voice processing error and the call ends. Runtype records the run as failed.

Usage and runs

Each turn counts toward your voice execution limits, like any other voice turn. Speech-to-text and text-to-speech are metered as for any other voice call. The run shows no model cost for the work that your agent does.

Each turn appears as a run with the surface type voice. For more information, see Working with runs.

Configurations that voice refuses

Runtype checks the agent and the conversation before the turn counts toward your voice execution limits. If voice can’t honor a setting, the caller hears Sorry, I encountered an error: followed by the reason. The following table lists each refused configuration and the reason that the caller hears:

Configuration	Reason the caller hears
`externalConfig.identityAssertion: true`	`Voice runtime admission cannot preserve: externalConfig.identityAssertion.`
`externalConfig.transport: "device"` (a local agent behind `runtype devices listen`)	`Voice runtime admission cannot preserve: externalConfig.transport "device".`
A conversation that belongs to an end user or a product tenant	`Voice runtime admission cannot preserve: an identity-scoped conversation record.`
`config.memory.enabled: true`, `config.sandbox.enabled: true`, or any `config.tenancyStrategy` or `config.artifacts`	`Voice runtime admission cannot preserve:` followed by each path that voice can’t honor.

For the entry points that can send an identity assertion, see Send the assertion to an external agent.

The following calls also don’t run an external agent:

A call that sends an identity proof: The connection fails with HTTP 422 and This agent type does not support voice identity proofs.
GPT-Live: speech.mode: "speech_to_speech" doesn’t run external agents. Use the pipeline mode.
A call that saves no conversation: An ElevenLabs call without a visitorToken or sessionId answers with Voice mode does not yet support external agents., as described in Start a voice call.

Next steps

Bring your own agent: register an external agent and implement a runtype-stream endpoint.
Build a custom voice interface: add call controls with the React hook.
Conversation history in an embedded widget: share one conversation between text and voice.
Working with runs: inspect each voice turn.