Architecture Layer: KV Cache Caveats
Prefix matching saves up to 90%; why switching tools invalidates the entire cache; sliding window vs chapter cache
THE QUESTION THIS PAGE ANSWERS
ANSWER FIRSTWhat is the key idea behind “Architecture Layer: KV Cache Caveats”?
Prefix matching saves up to 90%; why switching tools invalidates the entire cache; sliding window vs chapter cache
Read cost as a shape, not a single number. Break a request into input, output, retries, tools, and waiting time. The shape of usage usually tells you which design choice is expensive and where a smaller change can help.
Measure one real request before you optimize an imagined average.
A cheaper call that quietly creates more retries, latency, or review work.
An LLM is “Token pushes Token”: it doesn't care what you asked—only what came before. That means—if two requests share the same prefix, the model is recomputing the same work. KV Cache stores those intermediate results and reuses them on the next identical prefix: cheap storage for expensive live compute—“trading space for time.”
Example: you ask “2 + 4 = ?”, it gets 6. Ask “3 + 4 = ?”—prefix changed, start over. But “2 + 4 + 1 = ?”—prefix “2 + 4” is intact, so it starts from 6 and gets 7. As long as the prefix doesn't change, the cache can hit. Several hosted and self-hosted stacks offer some form of prompt or prefix caching, but support, retention, and discounts differ. Check the target provider's current documentation before you model the savings.
The previous request already cached the full prefix. Tap the three “this request” cases below and see hits (green) vs recomputes (red).
Looks like Token savings—it's a landmine
Some products mount tools by intent to save every Token: a weather tool for weather, nothing for small talk. The catch is the provider's request template: adding or removing a tools schema can change the prefix after the System Prompt; a template may also inject a default instruction block. Change tool state (on→off, A→B) and the Prompt-head prefix changes—hundreds of thousands of cached Tokens can go up in smoke.
Recommended: keep the System Prompt and tool schemas stable when cache reuse matters; or spend a few extra Tokens and keep a stable tool set mounted. System Prompt stability usually matters more than saving those schema Tokens.
In multi-turn and long-text products, many use a “sliding window” on long history—keep the last N turns, drop the rest. That's lazy, and it breaks. A sliding window is a FIFO queue: every scroll changes the prefix; the cache never hits.
A writing-assistant product that generates later text from earlier text started with a fixed 800-character rolling window—expensive and prone to “amnesia.” Later it switched to chapter cache: when the AI detects a topic shift (scene change, new chapter), it inserts a separator; the system decides what to compress into a summary and what to keep verbatim—better to keep the prefix stable and hit the cache than let the AI lossily compress on its own.
| Metric | Old (sliding window) | New (chapter cache) |
|---|---|---|
| KV Cache hit rate | ~10% (prefix always changing) | ~80% (prefix stability) |
| Logical coherence | Poor (frequent amnesia) | Good (summary + full chapters) |
| Token cost | High (recompute) | Low (cache reuse) |
| Question | Design decision |
|---|---|
| What must be “kept forever”? | Put it in the Stable zone as the cache prefix |
| What can be “compressed and archived”? | Replace originals with a summary; control window size |
| What needs “load on demand”? | Split by chapter / topic and mount dynamically |
| How to spot “compressible boundaries”? | Design a separator mechanism so the AI marks topic shifts |
Stable prefix → cache hit → save up to 90%. When picking an API, “does it support context caching?” is a must-check.
Don't switch tools dynamically: tool state changes rewrite the System Prompt template and invalidate the entire cache. Better keep the full set mounted.
Sliding windows kill the cache: replace with “Stable prefix + summary archive + chapter mount”—hit rate 10% → 80%.
Source: Adapted from the author's internal team share “AI Token Cost Engineering Strategy” hands-on section “03｜Architecture Layer.” For current cache semantics and pricing, compare the target provider's official docs, such as OpenAI prompt caching and Anthropic prompt caching; review “Token pushes Token” in LLM Fundamentals · Base Model.
The complete interaction cost of “What is KV Cache: prefix matching”
“An LLM is “Token pushes Token”: it doesn't care what you asked—only what came before.” is a reminder that AI cost is not one price multiplied by one call. Input, output, retries, tools, waiting time, and human cleanup together decide what a task really costs.
Find what the bill repeats
The key variables behind “Example: you ask “2 + 4 = ?”, it gets 6.” are usually repeated context, oversized output, retries after failure, and calls that do not produce useful progress. Removing wasted Tokens can reduce cost, latency, and concurrency pressure at the same time.
A cheaper call can make the whole workflow more expensive
Start with “Sliding windows kill the cache: replace with “Stable prefix + summary archive + chapter mount”—hit rate 10% → 80%” and keep a small table for input, output, retries, tools, and human review. Compare quality before and after optimizing instead of looking at one price in isolation.
From “What is KV Cache: prefix matching” to “Interactive Demo · Which Tokens hit the cache”
“What is KV Cache: prefix matching” grounds the problem in “An LLM is “Token pushes Token”: it doesn't care what you asked—only what came before. That means— if two requests share the same prefix, the model is recomputing the same work . KV Cache stores those intermedia…”. “Interactive Demo · Which Tokens hit the cache” then moves it toward “The previous request already cached the full prefix. Tap the three “this request” cases below and see hits (green) vs recomputes (red)”. Together, they show that the lesson is not just a conclusion to remember, but a claim with conditions.
Carry the judgment into the next situation
When analyzing cost, map the complete interaction first, then find repeated input, wasted output, and retries. A cheap individual call does not make the whole task cheap.
- “What is KV Cache: prefix matching”: An LLM is “Token pushes Token”: it doesn't care what you asked—only what came before. That means— if two requests share the same prefix, the model is recomputing the same work . KV Cache stores those intermedia…
- “Interactive Demo · Which Tokens hit the cache”: The previous request already cached the full prefix. Tap the three “this request” cases below and see hits (green) vs recomputes (red)
- “The closing point”: Stable prefix → cache hit → save up to 90%. When picking an API, “does it support context caching?” is a must-check
The final “The closing point” brings the discussion to “Stable prefix → cache hit → save up to 90%. When picking an API, “does it support context caching?” is a must-check”. The useful thing to carry forward is knowing which judgments must be revisited when input, scale, or risk changes.
I turned one judgment from this article into a small experiment I could run today. Knowing what to observe next is more useful than simply remembering the conclusion.
After reading this, I first looked for the conditions behind the idea instead of copying the method into a project. That order made the later trade-offs much clearer.
When this judgment reaches real work, which constraint should be added first? I am curious which step matters most between reading and the first practical attempt.
No discussion on this article yet.