When we build AI apps, we usually focus on making them fast, accurate, and easy to use. But as these system become more complex, the operating cost to run them can quickly skyrocket. This is exactly what happened to us.
In one of our projects, we built a chatbot using the LangGraph framework where a user message does not go through a single, fixed model call. Instead, an ” orchestrator” looks at the question and decides which specialist agents to run, and whether to run them in parallel or sequentially before providing a final answer to the user. A simple question might require only one agent, while a complex request might dispatch multiple agents and then combine the results.
The challenge was that each agent has its own system prompt, and these prompts are much larger than the user’s query. On average, a single agent sends roughly 6000 input tokens per request, excluding the user input. Because each agent carries large system prompt and the amount of context grows as the conversation continues, so the system can become expensive to operate.
We then started looking for ways to optimize the system and reduce its cost. We identified two main factors that make the system expensive
- The multiplier effect: A single user turn can invoke several agent calls. This means the same 6000 tokens prompt is paid multiple times for single question
- The snowball effect: As conversation/thread continues, the history travels with each request, this means that the prompt becomes larger with each turn, resulting in more input tokens per call which resulted in increasing the cost
How we reduced input token costs
We observed that the initial 6,000 tokens are the same on every call. The agent’s instructions do not change between turns, and earlier conversation turns do not change once they are done. But the provider has to read and consume all of it again on each request. That repeated processing was the cost that we wanted to remove.
The answer was prompt caching, where the provider checks the current request against the prompts it has seen earlier. The underlying engine processes the input from the first token and stops where the two prompts differ. The part that matches is considered a cache hit and billed at a lower rate.
The first request pays full price and gets stored. From the second request onwards, matching tokens at the front are billed at a cheaper rate. Because the match stops at the first difference it hits, stable content must always be kept at the front, and anything that changes must be placed after it.
If you accidentally put a changing value (like a timestamp) at the very top of your prompt, the cache engine will fail on the first line, forcing you to pay the full price for the entire 6000 token manual all over again.
The catch- why standard caching fails
The difficulty specific to this system is that the application does not know in advance which agent will answer. One request might be sent to agent 1, another to agent 2, and another to agent 3.
Every agent has a unique prompt. If the active agent changes from one request to the next, the model will not match the previous request’s prefix because that previous request belonged to a different agent.
Therefore, it was important for us to have an explicit caching mechanism that we can handle on our end for these providers. If we just construct static prefix and assume the providers will perform implicit caching from their end, it wouldn’t work that way.
We have to decide which part of the prompt stays fixed for an agent, keep it at the start of the input in the same order every time, and tell the provider to cache it. Each provider exposes caching differently, so here is what we did for OpenAI and Claude.
OpenAI (provider caching needed application-level direction)
OpenAI already does implicit caching on supported models, where OpenAI chooses matching prefix implicitly and reuses the matching prefix as the cached prefix. But this works only when all requests received by the model naturally share the same beginning.
But the implicit caching alone was not a good fit for a multi-agent workflow, because the selected agent can change from one request to the next. When the agent changes, its system prompt and tool definitions change too. With change in agent the differences occur at the beginning of the input, and with the change in prefix model invalidates the implicit cache.
The solution was to give OpenAI an explicit routing identity through prompt_cache_key. This key helps requests with the same prompt shape reach the same cache. It improves the chance of a hit, but it does not force a hit and does not replace the exact-prefix requirement.
Note:
In our implementation, OpenAI caching is controlled through the cache key. OpenAI still manages the prefix breakpoints (that is, calculating the matching prefix between the requests and validating how much input can be cached).
Two keys for two kinds of reuse
We developed two distinct key formats for our implementation.
- One key is the global level key, which includes {agent_name} as the cache key. This helps the model read the matching prefix from the cache even for newer requests, saving money.
- The other key is the conversation specific cache key, which will be helpful when the conversation gets bigger and the model can read the previous conversation from the cache. For this, the cache key is {agent_name}:{conversation_id}.
Prompt order made the keys useful
When the prompt is poorly organized, just creating prompt cache keys won’t save the money. As a result, we focused on the input that we are giving the model. Hence, we have divided the inputs into two categories (static and volatile) and then reorganized the input so that all the static input is kept at the beginning before being passed to the model.
Dynamic context belongs after the conversation because it can change on every model call, including retries and tool loops. If the dynamic context is inserted before the history, one changed keyword invalidates the entire conversation prefix. When dynamic data is placed last, the provider can still reuse the static prompt and all unchanged earlier turns. Re-appending it at the true end also ensures the model sees the freshest runtime information after tool activity.
Retention: how long the cache stays warm
A cache entry is only useful while it survives. OpenAI controls this through a retention setting on the request.
- Older supported models use prompt_cache_retention, where the extended 24h policy keeps a prefix eligible for reuse far longer than the short in-memory default.
- GPT-5.6 and later replace that field with prompt_cache_options.ttl, which sets a minimum lifetime instead of a storage policy.
The system requests extended retention for the model families that support it, so a returning conversation can still hit a warm prefix rather than paying full price again.
OpenAI cost overview
Rates below are for GPT-5.4 as a demo example, for inputs up to 272,000 tokens. A cached read is one tenth of the regular input price, and this model generation has no separate cache-write charge.
| Token type | Rate(USD / 1M tokens) | Relative to input |
| Regular input | $2.50 | 1x |
| Cached input (read) | $0.25 | 0.1x |
| Cache write | No separate charge | n/a |
| Output | $15.00 | n/a |
What the price difference looks like
Take one agent whose prompt is about 6,000 input tokens, where roughly 80% of it (about 4,800 tokens) is the stable prefix. Once the cache is warm, those 4,800 tokens can be reused. Across a ten-request conversation, the drop in the token fees is clear.
(Note: OpenAI’s current behavior differs by model family. GPT-5.6 and later can use explicit breakpoints, charge 1.25 times the normal input rate for cache writes, and charge 0.1 times for cache reads. Earlier models use implicit breakpoints, generally have no separate write charge, and use model-specific cached-input rates).
Anthropic (explicit breakpoints made the boundary visible)
The same prefix principle is used in Anthropic, but a direct control is exposed. In order to implement explicit caching in Anthropic, we must set a cache break point. This means, we inform the Claude models that the input is cacheable up to this breakpoint. This control is necessary because Anthropic features a distinct cache-write cost alongside its read discounts.
When anthropic model receives the request, it determines whether the prefix that ends at the breakpoint is already present. If prefix is found, then then it will read at the cache-read rate. In the event of a miss, the chosen cache-write rate is used to process and write that prefix. Therefore, we must exercise caution while establishing the breakpoint with anthropic because if it is done incorrectly, the cost will exceed the input cost.
On an early request, there may be no assistant response before the current user message. In that case, the stable system prompt receives the breakpoint. On a later request, the breakpoint moves to the assistant message immediately before the current real user request.
Because the cache is cumulative, that one boundary already covers the system prompt, and all the prior conversation turns. The current user message and synthetic dynamic context remain uncached below the breakpoint line, giving us full read-time operational flexibility without risking our cost savings.
Retention: the 5-minute and 1-hour choices
Unlike OpenAI, Anthropic prices retention differently for the two lifetimes. The default cache lifetime is five minutes. A five-minute write costs 1.25 times (varies from model to model, the prices shown are for Claude-sonnet 5) the base input price. A one-hour write costs 2 times the base input price. Cache reads cost 0.1 (varies from model to model) times the base input price for the models used in this design.
Each successful read refreshes the lifetime, measured from the start of the request, so an active conversation keeps the entry warm on its own.
Anthropic cost overview
Rates below are for Claude Sonnet 5 as a representative example. The write charge is the extra cost paid the first time a prefix is stored, and it is the only place where the 5-minute and 1-hour choices differ in price.
| Operation | Rate (USD / 1M tokens) | Relative to input |
| Regular input | $2.00 | 1x |
| Cache read (hit) | $0.20 | 0.1x |
| 5-minute cache write | $2.50 | 1.25x |
| 1-hour cache write | $4.00 | 2x |
| Output | $10.00 | n/a |
The five-minute option was selected because interactive multi-agent conversations are bursty. Once a user starts a task, follow-up model calls, tool loops, and the next user turn usually arrive close together, well inside five minutes. Every hit refreshes the window, so an active conversation stays warm without paying the one-hour write premium.
The one-hour option is useful when reuse is sparse but predictable, such as a long pause between repeated requests. The one our pattern is not impactful in our current implementation. Paying 2 times for every new cache write would increase the miss penalty while providing little extra value during a long conversation. The choice is a workload decision, not a claim that five minutes is always better.
Explicit controls side by side
Both providers cache prefixes, but the application signals reuse differently. OpenAI routes by key, Anthropic marks breakpoints. The prompt order rules are identical for both.
| OpenAI | Anthropic | |
| Explicit control | prompt_cache_key (+ retention) | cache_control: { type: “ephemeral”, ttl: 5m/1h } |
| What it does | Routes requests to the right cache pool per agent/conversation | Marks exactly where the reusable prefix ends |
| On each call | Lookup by key -> match prefix -> read or write | Check prefix to breakpoint -> read or write |
| Hit billing | ~0.1x input (cache read, varies by model) | Cache read rate, varies by Claude model |
| Miss billing | Full input or model specific write (varies by model) | Cache write rate, varies by Claude model (5m / 1h TTL) |
| Lifetime | 24h extended retention (supported models) | 5m or 1h ephemeral, refreshed on each read |
| Multi-agent isolation | Different key per agent | Different breakpoints per agent prompt shape |
What made the savings durable
The largest improvement did not come from one provider flag. It came from treating prompt structure as part of the caching design. Agent identity determines routing. Stable instructions stay at the front. Conversation history grows by appending rather than rewriting. Dynamic context stays at the end. Provider usage fields are recorded so cache reads, cache writes, hit ratios, and cache aware cost can be inspected instead of assumed.
OpenAI and Anthropic expose different controls, but the rule is the same: keep the expensive, reusable prefix stable and make the changing suffix as small as possible. Once that boundary is defined, repeated input stops being an unavoidable cost on every agent call.
We used OpenAI and Anthropic in this system. Gemini catches in a different way. If you want to see how that works, you can check out this blog: Gemini-Context-Caching
About Author
Mudadla Siva Sai is a software engineer specializing in building autonomous, context-aware applications and Multi-Agent Systems. Certified with the GitHub Copilot GH-300 credential, he focuses on optimizing complex AI workflows, mitigating hallucinations, and exploring emerging Agent-to-Agent (A2A) architectures
