Prompt caching

Runtype automatically caches the stable parts of your prompts with supported model providers so that repeated executions run faster and cost less. There is nothing to configure. When caching is likely to help, Runtype turns it on for you, and when it is not, Runtype leaves it off.

How prompt caching works

Most prompts share a large, unchanging prefix from one call to the next: the system prompt, the tool definitions, and the earlier turns of a conversation. Providers that support prompt caching can store that prefix after the first request and reuse it on later requests instead of reprocessing it every time.

Runtype composes your prompt, so it already knows which parts are stable and which change on every call. It marks the stable prefix as cacheable and lets the provider serve it from cache on the next request. Cached input is billed at a large discount compared to fresh input, and it reaches the model faster, which lowers the time to the first token.

Prompt caching is available today for the Anthropic model family (Claude models) and the OpenAI model family (GPT and o-series models), including when those models are reached through platform keys. Support for more providers is rolling out over time. When a model or provider does not support caching, or has not yet passed our billing verification, your requests run exactly as before with no change in behavior and cached input is billed at the regular input price.

Automatic tuning

Runtype also watches how caching performs in practice, not just how it looks up front. Writing an entry to the cache carries a small premium, so a workload that keeps writing entries without ever reading them back costs more than it should. When that happens repeatedly for a workload (many cached requests in a row with almost no cache reads over a day), Runtype pauses caching for that workload for about 24 hours and then automatically tries again.

You do not need to do anything. If the pause was caused by a changing value early in the prompt (see Writing cache-friendly prompts below), fixing the prompt or publishing a new version of the flow or agent restores caching immediately. This is also the most common reason a workload that previously showed cache reads temporarily shows none: the pause removes the write premium while it is in effect, and normal caching resumes on its own.

When caching helps

Caching pays off when the same prefix is reused across several requests. Runtype enables it automatically in these cases:

• Agent loops. An agent that runs multiple turns re-sends its system prompt, tools, and growing history on each turn. Every turn after the first can read the earlier prefix from cache.
• Conversations with history. Chat surfaces and any request that carries previous messages reuse the conversation prefix as it grows.
• Batch and eval fan-outs. When you run a batch or an eval over one flow or agent version, every record shares the same system prompt and tool definitions. That shared prefix is cached once and read by the rest of the run.

Single-shot prompts with no history are not marked for caching. On providers that charge a cache write premium (such as Anthropic), writing an entry only pays off when it is read again, so Runtype does not cache a prompt that will only be sent once. Providers with automatic caching (such as OpenAI) may still cache repeated prompts on their own. Cache writes are free on those providers, so this never adds cost.

What you need to do

Nothing. There are no API changes, no cache keys to manage, and no settings to turn on. Caching happens at the model provider and is handled entirely by Runtype.

There is one escape hatch. If you have a latency-sensitive workload of unique prompts that will never benefit from caching, you can force caching off for a specific request by setting cache to false in the options of a dispatch or execute request:

1
{
2
  "options": {
3
    "cache": false
4
  }
5
}

This is rarely needed. Leave caching on unless you have measured that it does not help a particular workload.

Writing cache-friendly prompts

Caching only reuses a prefix when that prefix is byte-for-byte identical from one request to the next. A value that changes on every run, placed early in the prompt, changes the whole prefix and prevents later requests from reading anything from cache.

The most common culprits are the current date ({{_now}}) and per-execution variables ({{_execution.*}}). Put volatile content like this late in the prompt, for example in the user message, rather than near the top of the system prompt. Keeping the system prompt and tool definitions stable lets Runtype cache the largest possible prefix across requests.

Runtype checks for this at save time. If a flow’s system prompt embeds a changing value like {{_now}} or {{_execution.*}} early, you will see a non-blocking validation advisory suggesting you move it later. The advisory never blocks saving. It is a hint that your prompt could cache better with a small change. See Flow validation warnings for how these advisories appear.

Seeing cache activity

Cache read and cache write token counts appear in the step detail for each prompt step (open a run and expand a step to see its token breakdown). A high cache read count on later turns of a conversation or across a batch run is the sign that caching is working for that workload.

Aggregate cache activity appears on the Usage page (Settings → Usage). When your account has cache activity in the selected window, a Prompt cache card shows the estimated savings, the overall cache hit rate, and total cache read and write tokens. Each row in Cost by AI Model also shows what share of that model’s input tokens was served from cache. The savings figure is an estimate at current catalog rates: cache reads are billed at a discount, and on providers with a write premium (such as Anthropic) the premium is netted against the savings.

Prompt caching is not response caching

Next steps

• Available AI models to see which models you can use.
• Usage data to track how caching affects your spending.
• Working with logs to trace a run and open its step detail.
• Flow validation warnings to understand the cache-friendliness advisory.