Prompt caching

Prompt caching reuses a stable prefix of your prompt across requests. Runtype manages provider-specific cache settings, so you do not configure cache keys or breakpoints. A cache read can reduce input cost and time to first token.

How prompt caching works

Your prompt often starts with stable content, such as the system prompt, tool definitions, and previous conversation turns. A provider that supports prompt caching stores that prefix after the first request and reuses it on later requests.

Runtype composes your prompt and identifies stable segments. It marks reusable prefixes for providers that require cache markers and adds routing hints for providers that use automatic caching. The provider reports cache reads and cache writes with the model response.

Runtype recognizes prompt-cache usage for the Anthropic, OpenAI, and DeepSeek model families. It uses verified cached-input rates for usage estimates for these families. Anthropic and OpenAI support includes calls made with platform keys. For other providers, Runtype uses the regular input rate when it does not have a verified cached-input rate.

Automatic tuning

Runtype monitors requests that carry prompt-cache markers for each Flow or Agent version. When a 24-hour window contains at least 20 such requests, Runtype checks the read rate. If fewer than 5% produce cache reads, Runtype suppresses new prompt-cache markers for that workload. The suppression lasts 24 hours and avoids repeated cache-write premiums for a prefix that does not produce cache reads.

To restart caching sooner after you fix a volatile system prompt, publish a new Flow or Agent version. A new version uses a new workload key. Otherwise, caching resumes after the 24-hour suppression expires. For prompt placement guidance, see Writing cache-friendly prompts.

When caching helps

Prompt caching helps when you send the same prefix more than once. Runtype enables provider caching automatically for these request patterns:

• Agent loops: Your Agent resends its system prompt, tools, and growing conversation history on each turn. Later turns can read the earlier prefix from cache.
• Conversations with history: When you send previous messages, each request reuses the conversation prefix that remains unchanged.
• Batch and Eval fan-outs: When you run a batch or Eval over one Flow or Agent version, the records share the same system prompt and tool definitions. The first record warms the shared prefix, and later records can read it from cache.
• Repeated single-shot requests: When you repeat a request with a stable system prompt or tools, the provider can reuse that prefix. Runtype does not add cache markers when the request has no stable system prompt or tools.

A one-time request has no cross-request cache read. For providers that charge a cache-write premium, Runtype avoids speculative cache markers when a request has no reusable prefix. Providers with automatic caching can still reuse a repeated, byte-identical prefix.

What you need to do

Do not configure prompt caching. You do not manage cache keys or cache breakpoints.

To disable prompt caching for a dispatch request, set cache to false in options as shown in this JSON fragment:

1
{
2
  "options": {
3
    "cache": false
4
  }
5
}

To disable prompt caching for an Execute Agent request, set the top-level cache field to false as shown in this JSON fragment:

1
{
2
  "cache": false
3
}

Use this opt-out when requests must not share provider cache state or when you compare cached and uncached behavior.

Writing cache-friendly prompts

To get cache reads, keep the reusable prefix byte-for-byte identical across requests. A value that changes on every run changes the prefix that follows it and prevents later requests from reading that prefix from cache.

Move per-run values into the user prompt or a later step. Do not place {{_now}}, {{_execution.*}}, or {{_schedule.*}} in the system prompt. The cache planner treats any of these variables in the system prompt as volatile, regardless of their position.

When you save a Flow, Runtype reports a non-blocking CACHE_VOLATILE_SYSTEM_PROMPT recommendation for these variables. Move the changing value to the user prompt or a later step. The recommendation does not block saving or change Flow behavior. For more information, see Flow validation warnings.

View cache activity

To inspect one run, open Logs, select the run, and open a prompt step in the Step inspector. The step detail shows cache read and cache write token counts when the provider reports them. A nonzero cache read count confirms that the provider served part of the input from cache.

To review aggregate activity, open Settings > Usage. When the selected period includes cache activity, the Prompt cache card shows estimated savings, cache hit rate, cache read tokens, and cache write tokens. The Cost by AI Model table shows each model’s share of input tokens served from cache. Runtype calculates the savings estimate from catalog rates and subtracts cache-write costs when the provider charges them.

Prompt caching and response caching

Use this distinction when you choose a caching strategy.

Next steps

Use these guides to continue:

• Available AI models: choose a model that fits your workload.
• Usage data: review cache activity and estimated savings.
• Working with logs: inspect cache token counts for a run.
• Flow validation warnings: understand the cache recommendation.