Skip to navigation
Integrations

Braintrust (tracing)

Send a trace of each model call in your Runtype runs, including the prompt, output, and token usage, to a Braintrust project. Your Braintrust connection doesn’t replace or change the built-in Runtype logs.

Before you begin

You need the following:

  • A Braintrust account and a Braintrust API key.
  • A Team plan or higher. For more information, see Billing and plans.
  • In an organization, the organization admin role to change or remove the connection.

Run Investigator explanations requested with Explain this run are an exception: their analysis model calls are not forwarded to Braintrust, to avoid exporting the trace excerpt again. Normal model usage accounting still applies.

Runtype sends traces to Braintrust’s hosted service at braintrust.dev. Self-hosted Braintrust deployments aren’t supported.

Connect Braintrust

To connect a Braintrust project, follow these steps:

  1. In Runtype, go to Settings > Integrations.
  2. In Braintrust, enter your API key in Braintrust API Key.
  3. Optional: Enter a project name in Project Name (optional). Runtype sends traces to this Braintrust project. If you leave the field blank, Runtype uses the Runtype project.
  4. Optional: To mask email addresses, phone numbers, card numbers, and government IDs in traces, select Personally identifiable information (PII).
  5. Click Save Key.

When you save, Runtype checks the key with Braintrust. If Braintrust rejects the key, the save fails. To disconnect Braintrust, click Remove Key.

In an organization, the connection belongs to the organization, and runs use it no matter which member added it. The audit trail records the member who made each change. Runtype encrypts your API key at rest.

Connect with the API

To connect Braintrust from a script, send a POST request to /v1/provider-keys with an API key that has the INTEGRATIONS:WRITE permission. The following request connects the customer-support project and masks personal data in traces:

cURL
curl -X POST https://api.runtype.com/v1/provider-keys \
  -H "Authorization: Bearer YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "provider": "braintrust",
    "name": "Braintrust",
    "apiKey": "YOUR_BRAINTRUST_API_KEY",
    "isDefault": true,
    "settings": {
      "projectName": "customer-support",
      "redactFromTraces": ["pii"]
    }
  }'

Replace the following:

  • YOUR_API_KEY: your Runtype API key
  • YOUR_BRAINTRUST_API_KEY: your Braintrust API key

The request uses the following settings:

  • projectName: The Braintrust project that receives traces. If you omit it, Runtype uses the Runtype project.
  • redactFromTraces: Set to ["pii"] to mask email addresses, phone numbers, card numbers, and government IDs before Runtype sends a trace. Omit it to send trace content unmasked.

If your organization has more than one active Braintrust connection, Runtype sends traces only to the connection marked as the default. Set isDefault to true on the connection you want. If no connection is the default, Runtype doesn’t send traces. Saving in Settings > Integrations always marks the connection as the default.

To rotate the key or change the settings, send a PATCH request to /v1/provider-keys/YOUR_CONNECTION_ID with the fields to change. To remove the connection, send a DELETE request to the same path. For the full request and response schema, see Create a provider key.

Verify the connection

To confirm that traces arrive, follow these steps:

  1. Run a flow or an agent that makes at least one model call.
  2. In Braintrust, open the project you connected. A runtype-execution span appears for each model call in the run.

Troubleshooting

The following problems are the most common:

  • Braintrust rejected this API key. Braintrust didn’t accept the key when you saved it. Copy the key again from Braintrust and save it.
  • Bring Your Own Key requires a Team plan or higher. Your plan doesn’t include connections to outside services. Upgrade your plan, then connect again.
  • Traces don’t appear. Check that the project name matches your Braintrust project and that the key is still valid in Braintrust. If Braintrust was unreachable when you saved, Runtype accepted the key without checking it, so an invalid key can save successfully. If your organization has several connections, choose one in Braintrust connection and click Save Key to make it the default. A failed export doesn’t fail the run, so a run can succeed while its trace is missing.

What gets traced

Runtype traces model calls in the following places:

  • Flow runs from the dashboard, the API, and schedules
  • Agents
  • Product surfaces
  • Batches
  • Evals

Runtype doesn’t trace model calls that you make outside Runtype, or calls from a self-hosted @runtypelabs/runtime deployment. Runtype sends traces in the background, so a Braintrust outage doesn’t slow down or block your runs.

What each trace contains

Each model call creates a runtype-execution parent span with a child span of type llm. The parent span holds the user, organization, flow, and execution IDs. The child span holds the input, output, provider, model, finish reason, token usage, and latency. If a provider call fails, the child span records the error and any partial output.

Trace content limits

Trace export doesn’t change your run results. Before export, Runtype removes secret values from the trace, and masks personal data if you turned on PII redaction. If the trace is still larger than 1,000,000 bytes, Runtype replaces the input, output, and error with a placeholder and drops custom metadata. The provider, model, timing, and token usage remain. If the trace is still too large, Runtype doesn’t export it.

Next steps