We built an enterprise legal document chat application running on Vertex AI recently using Gemini and Google’s Agent Development Kit (ADK). We faced a production challenge while building a multi-turn document chat. Users, there, uploaded a single large PDF (like a 100-page contract, ~43,000 tokens) and asked a series of follow-up questions. Apparently, the interaction looked like one fluid conversation where each turn was a stateless API request. But each user’s question got translated to an independent, stateless LLM API request under the hood.
This meant we were resending the massive document every single turn. It resulted in high latency, soaring costs, and architecture that was difficult to debug. Optimization was the only way out.
Per-Session Economics (8-Turn Session)
| Approach | Cost / Session | Cost / Question | Savings / Session |
|---|---|---|---|
| No Cache (Resend Document Every Turn) | ~$0.93 | ~$0.12 | Baseline (0%) |
| Explicit Cache (Document Prefix Cached) | ~$0.44 | ~$0.05 | ~$0.49 (53%) |
We solved the problem by implementing Gemini Explicit Context Caching. In a representative 8-question session, we handled 800 sessions per month (approximately 6,400 API calls per month, or about 27 sessions per day) and reduced costs by approximately 53% (from $0.93 to $0.44 per session), resulting in savings of around $400 per month. But the technical implementation revealed a critical engineering lesson. Caching everything isn’t the solution; the real breakthrough was learning exactly what belongs in the cache and what must absolutely stay dynamic.
This post shares our problem, the architecture that almost backfired, the solution we shipped, and the production takeaways that matter for performance and cost.
Context: The Two Faces of Caching
Dashboard document chat seems straightforward: upload a contract, ask questions, and keep going. But behind the screen, every Gemini call requires two distinct components:
- Stable Context: The unchanging data, including the uploaded document, system instructions, and role prompts.
- Dynamic Context: The ever-evolving conversation history and the user’s latest question.
Naïve implementations treat these identically, leading to massive data redundancy. To optimize, Gemini offers two caching modes:
- Implicit Caching: Automatic, probabilistic, with no storage fees but also no guaranteed hit.
- Explicit Caching: Developer-controlled creation of a CachedContent resource, which persists for a defined TTL and is referenced by name.
For our workload, implicit caching wasn’t sufficient as users pause between complex legal analyses naturally. We needed predictable, guaranteed reuse of a large PDF across an entire session, making explicit caching the only viable choice.
The Mistake That Can be Made
When you first enable explicit caching, your instinct is simple: if document context benefits from caching, conversation history probably does too.
It does not.
Chat history grows every single turn. If you include this expanding history inside the CachedContent resource, your cache becomes stale instantly. You find yourself constantly recreating the cache, paying full cache creation rates again, and stacking additional storage charges. In our cost estimations, this “cache-everything” path quickly became more expensive than using no cache at all.
Caption: A critical distinction: Caching history means constant, expensive cache recreation (wrong). Caching only the stable document prefix enables seamless reuse (right).
Key production takeaway: Cache what is large and stable. Never cache what grows every turn.
What We Actually Built: Prefix-Suffix Separation
We designed our request architecture to treat every turn as a combination of a stable prefix and a dynamic suffix.
| Part | Examples | Where It Lives |
|---|---|---|
| Stable Prefix | PDF Contract, System Instructions, Role Prompts | Explicit CachedContent |
| Dynamic Suffix | Conversation History, Current Question | Request body (normal prompt) |
High-Level Workflow:
- Ingest: User uploads a document $\to$ stored in GCS.
- Initialize: App creates CachedContent once (PDF + system instruction, with a 15-minute default TTL).
- Identify: The session stores the cache_name and a fingerprint of the stable system prompt.
- Chat: For follow-up messages, the app sends only questions and history, not the PDF.
- Inject: Google ADK injects the cached_content via a before_model_callback.
- Cleanup: On explicit session end, we delete the cache to avoid incurring idle storage charges.
Caption: Architectural view of a request: The stable prefix is fetched instantly from CachedContent, while history and the new question are sent dynamically in the request body.
Engineering watchpoint: If the system instruction is already inside the cache, your code must not send it again. This duplicate instruction was an easy “foot-gun” that could cause model confusion and increased tokens. Conversation continuity still works perfectly; history stays as dynamic input.
Cost Breakdown: The View from production
The “discounted cached reads” headline is appealing, but it’s incomplete. Production explicit caching is driven by four key cost components:
| # | Charge | What it is |
|---|---|---|
| A | Cache Creation | Full input rate on the first turn when you build CachedContent. |
| B | Cache Storage | Tokens per hour (charged while the cache lives). |
| C | Cached Input | Deeply discounted reads on turns 2+ (this is the savings part). |
| D | Normal I/O | History + Current Question + Model Answer tokens (dynamic cost). |
If your internal dashboard only tracks per-message inference tokens (C+D), your analytics will look significantly better than what finance actually sees on the GCP bill. You must account for A and B.
Here is an illustrative comparison of the costs for our sample 8-turn session with a ~43k-token document.
| Approach | Illustrative 8-Turn Session Cost |
|---|---|
| No Caching (Send Doc Every Turn) | ~$0.93 |
| Explicit Caching (Document Only) | ~$0.44 |
| Document + History in Cache | Often worse than no cache |
Caption: Visualizing the 53% savings: Explicit caching requires an upfront creation cost and storage (multi-color bar components), but the drastically cheaper “Cached Input Rate” (seen in Turns 2-8) results in a lower overall session cost compared to sending the document every turn.
TTL is a Product Decision, Not Just a Config Knob
Setting a very short TTL (e.g., 2 minutes) looks cheap on storage. But real users read an answer, pause to think, synthesize the information, and then type again. If the cache expires while users are still processing the information, it incurs the cost of recreating the document cache, negating any savings.
For our workload, 15 minutes with explicit deletion at the session end was the ideal balance between human pauses and storage cost. We now treat the “cache recreate rate” (the frequency at which we miss the cache during an ongoing session) as a vital health signal. Spikes usually mean our TTL is too aggressive for the user’s behavior, or our prompt fingerprinting is busted.
Conclusion
Context caching was more than just a token of optimization; it forced a cleaner architectural separation between stable document state and dynamic conversation state.
LLM features are often marketed as “easy to turn on.” But the real engineering work is deciding where reliability and determinism belong and where clever optimizations result in higher costs. Our definitive rule of thumb for multi-turn document chat:
Cache the stable prefix. Keep the dynamic conversation outside.
