Skip to navigation
Settings

Prompt caching

Prompt caching reuses a stable prefix of your prompt across requests. Runtype manages provider-specific cache settings, so you do not configure cache keys or breakpoints. A cache read can reduce input cost and time to first token.

How prompt caching works

Your prompt often starts with stable content, such as the system prompt, tool definitions, and previous conversation turns. A provider that supports prompt caching stores that prefix after the first request and reuses it on later requests.

Runtype handles each provider’s caching requirements for you. The provider reports cache reads and cache writes with each response.

Prompt caching does not store model responses. The model runs on every request and returns a fresh response. Caching reduces only the cost and latency of processing the input.

Provider caches have their own retention rules. A cache opt-out does not delete earlier cache entries or saved application data. For the relationship between caching, inference ZDR, and storage, see Data retention.

Runtype recognizes prompt-cache usage for the Anthropic, OpenAI, and DeepSeek model families, including Anthropic and OpenAI calls made with platform keys. Usage estimates for these families apply the provider’s discounted cached-input rate. For other providers, usage estimates price cached input at the regular input rate when Runtype has no verified cached-input rate for the model.

When caching helps

Prompt caching helps when you send the same prefix more than once. Runtype turns on provider caching automatically for these request patterns:

  • Agent loops: Your Agent resends its system prompt, tools, and growing conversation history on each turn. Later turns can read the earlier prefix from cache.
  • Conversations with history: When you send previous messages, each request reuses the conversation prefix that remains unchanged.
  • Batch and Eval fan-outs: When you run a batch or Eval over one Flow or Agent version, the records share the same system prompt and tool definitions. Later records can read that shared prefix from cache.
  • Repeated single-shot requests: When you repeat a request with a stable system prompt or tools, the provider can reuse that prefix. Runtype does not request caching when the request has no stable system prompt or tools.

A one-time request has no cross-request cache read. For providers that charge a cache-write premium, Runtype does not request caching when a request has no reusable prefix. Providers with automatic caching can still reuse a repeated, byte-identical prefix.

Batch warm-up

For a batch with more than one record, Runtype runs one record first to write the shared prefix to the cache, and then runs the remaining records in parallel. This adds the latency of one record before the batch fans out.

To start all records at once, set skipCacheWarm to true in the batch options object. Use this option when records share no stable prefix or when throughput matters more than cache reads. The same option is available in the TypeScript SDK BatchBuilder and in the execution options of a schedule. For more information, see Running flows in batch.

Write cache-friendly prompts

To get cache reads, keep the reusable prefix byte-for-byte identical across requests. A value that changes on every run changes the prefix that follows it and prevents later requests from reading that prefix from cache.

Follow these guidelines when you write prompts:

  • Put large, stable context, such as style guides, schemas, and examples, in the system prompt. Vary only the user prompt.
  • Give steps that perform the same task an identical system prompt, so they share one cache entry.
  • Move per-run values into the user prompt or a later step. Do not place any _now, _execution, or _schedule variable, such as {{_now}} or {{_execution.id}}, in the system prompt. If any of these variables appears anywhere in the system prompt, the provider cannot reuse the system prompt from cache.
  • Expect other variables in the system prompt, such as {{_user.id}} or a previous step’s output, to create a separate cache entry for each distinct value. They do not stop caching.

For the full list of system variables, see Flow variables and templates.

When you save a Flow, Runtype reports a non-blocking CACHE_VOLATILE_SYSTEM_PROMPT recommendation for per-run variables in a system prompt. The recommendation does not block saving or change Flow behavior. Runtype checks top-level prompt steps only. Prompt steps inside a conditional step are not checked. For more information, see Flow validation warnings.

Turn off prompt caching

Prompt caching needs no configuration. To turn it off for one request, set cache to false. Use this opt-out when requests must not share provider cache state or when you compare cached and uncached behavior.

For a POST /v1/dispatch request, set cache in the options object as shown in this JSON fragment:

{
  "options": {
    "cache": false
  }
}

For a POST /v1/agents/{id}/execute request, set the top-level cache field as shown in this JSON fragment:

{
  "cache": false
}

The TypeScript SDK accepts the same field: pass cache: false to the agents.execute method, or include it in the options you pass to the withOptions method of a FlowBuilder. The Runtype MCP server dispatch and execute_agent tools accept a cache parameter.

The opt-out applies per request. Batch, Eval, and schedule runs have no cache option. For these runs, use skipCacheWarm to skip the warm-up record.

View cache activity

To inspect one run, open Logs, select the run, and select a prompt step. The step details show cache read and cache write token counts when the provider reports them. A nonzero cache read count confirms that the provider served part of the input from cache.

To review aggregate activity, open Settings > Usage. The Estimated cache savings tile shows the estimated savings and the cache hit rate. The Prompt cache card shows cache read and cache write token totals. The savings estimate is net of any cache-write premium the provider charges. Both appear only when the selected period contains cache activity. In the Cost by AI model table, a percent-cached badge appears next to each model that served input from cache. For more information, see Usage data.

To check caching from code, call GET /v1/runs/{executionId} and read the cacheReadTokens and cacheWriteTokens fields. Both are subsets of inputTokens.

Troubleshoot missing cache reads

If the cache read count stays at zero, check the following causes:

  • The stable prefix is short. Providers cache only prefixes above a minimum length. Runtype does not request caching for a prefix it estimates is under that minimum, so a short system prompt with few tools often does not cache.
  • Requests are too far apart. A cache entry typically expires about 5 minutes after its last use. Requests with the same prefix that arrive further apart do not read from cache.
  • The system prompt changes on every run. Move per-run variables out of the system prompt, as described in Write cache-friendly prompts.
  • The request turned caching off. Check whether the request sets cache to false.
  • The provider is not supported. Runtype recognizes cache usage for the Anthropic, OpenAI, and DeepSeek model families.
  • Runtype paused caching for the workload. For more information, see Why caching stops for a workload.

Why caching stops for a workload

Runtype tracks cache reads for each Flow version and each Agent. If at least 20 requests in a 24-hour window request caching and fewer than 5% of them produce a cache read, Runtype stops requesting caching for that workload for 24 hours. This avoids repeated cache-write premiums for a prefix that is rarely reused. Providers with automatic caching can still reuse an identical prefix during the pause.

After you fix the prompt, caching restarts as follows:

  • Agent: Caching restarts when the 24-hour pause ends. Publishing a new Agent version does not reset the pause.
  • Flow: Publish a new Flow version to restart caching immediately. Otherwise, caching restarts when the 24-hour pause ends.

Next steps

Use these guides to continue: