Caches: The Invisible Discount on Your AI Bill
KV Cache and semantic cache are the same move: don't recompute. Drag the hit-rate slider and watch the bill change live—the underlying principle of cost optimization in the Harness core part
THE QUESTION THIS PAGE ANSWERS
ANSWER FIRSTWhat is the key idea behind “Caches: The Invisible Discount on Your AI Bill”?
KV Cache and semantic cache are the same move: don't recompute. Drag the hit-rate slider and watch the bill change live—the underlying principle of cost optimization in the Harness core part
Make the claim earn its place. Use this page as a decision aid, not a definition to memorize. Connect the idea to one real task, one observable result, and one failure that would change your mind.
Write one question you could answer with evidence after trying this idea.
A conclusion that sounds complete but leaves the key assumption untested.
You once computed “37 × 89 = 3293.” Next day someone asks again—do you redo the long multiplication? No, you just give the answer. A cache is the computer's “just give the answer”: store results by key in last lesson's hash table; same key next time, one-step fetch. Hash tables handle “where & how to find”; caches handle “what's worth keeping.” Together they're the full “trade space for time.”
Every time a large model generates a token, it “looks back” at all prior tokens and computes attention intermediates for each. The key: if the prefix is identical, those intermediates are identical—so why recompute them in round two? Each square below is a token. Play “Round 1,” then “Round 2.” Watch for how fast the green squares appear in round two: they weren't computed—they came from cache.
Round 1 conversation
System prompt + question ①—every token computed from scratchRound 2 conversation
Prefix (system prompt + all of round 1) unchangedRound-2 compute comparison (squares = tokens to compute)
KV Cache saves on “the same opening.” There's a fiercer move: same question—don't recompute the whole answer. A support bot gets asked “how do I return this?” 10,000 times a day—wordings vary, meaning doesn't. Treat “meaning” as the key (vector similarity; lesson ten), and on a hit return the stored answer with zero model calls. Drag the hit-rate slider—watch for the monthly bill.
⚠️ But—what if the return policy changes?
The cache still holds a standard answer generated under the old policy. The bot will keep serving it earnestly for days or weeks—wronger than no cache, and more confidently. Engineers say: the hard part of caching isn't storing—it's knowing when to invalidate (jargon: “cache invalidation,” one of CS's two hard problems). Common moves: TTL (e.g. expire in 24 hours), or purge related entries the moment policy updates. Design that step before any cache—or the money you save comes back as support tickets.
“Don't recompute” is everywhere. These four things you enjoy every day are the same structure underneath.
KV Cache
Standard kit for LLM inference: attention intermediates for prefix tokens are computed once. Without it, long chats simply don't run.
Semantic cache
Treat “question meaning” as the key; similar asks reuse the answer. High-volume support can cut call fees in half or more.
Browser cache
Images and styles download once and stay local; the second page load is instant—half of why browsing feels fast.
CDN
Store content ahead of time in the nearest POP so users everywhere feel like hitting a local server. Cache + geography—same move.
Why “Start with an everyday scene” depends on the operation
“You once computed “37 × 89 = 3293.” Next day someone asks again—do you redo the long multiplication?” makes the structure concrete. The useful comparison is not which name sounds more advanced, but how the data is arranged and how far the most common operation has to travel.
Read a structure through access and change
“Every time a large model generates a token, it “looks back” at all prior tokens and computes attention intermediates for each.” exposes a trade-off that is easy to miss: reading by position, looking up by key, adding at either end, inserting in the middle, and traversing relationships do not favor the same organization. A structure that is fast for one operation is not automatically fast for all of them.
- Cache = don't recompute : store results by key in a hash table, fetch next time—last lesson's structure starts earning money here
- KV Cache keys on prefix : fixed content at the prompt head, changing bits at the end—hit rate writes the bill
- Semantic cache is fiercer : same-meaning asks skip the model entirely; savings scale linearly with hit rate in high-volume scenes
Count scale and update frequency together
Use “Store content ahead of time in the nearest POP so users everywhere feel like hitting a local server.” as a boundary check. Write down the data size, the dominant operation, and the latency you can accept before deciding whether an AI-generated structure actually fits.
From “Start with an everyday scene” to “Interactive 1 · KV Cache: don't recompute the same prefix twice”
“Start with an everyday scene” grounds the problem in “You once computed “37 × 89 = 3293.” Next day someone asks again—do you redo the long multiplication? No, you just give the answer. A cache is the computer's “just give the answer” : store results by key in last…”. “Interactive 1 · KV Cache: don't recompute the same prefix twice” then moves it toward “Every time a large model generates a token, it “looks back” at all prior tokens and computes attention intermediates for each. The key: if the prefix is identical, those intermediates are identical —so why reco…”. Together, they show that the lesson is not just a conclusion to remember, but a claim with conditions.
Carry the judgment into the next situation
When you meet a new data structure, do not begin by memorizing its definition. Write down the most frequent operation, estimate scale and update behavior, and check whether the structure satisfies all three conditions.
- “Start with an everyday scene”: You once computed “37 × 89 = 3293.” Next day someone asks again—do you redo the long multiplication? No, you just give the answer. A cache is the computer's “just give the answer” : store results by key in last…
- “Interactive 1 · KV Cache: don't recompute the same prefix twice”: Every time a large model generates a token, it “looks back” at all prior tokens and computes attention intermediates for each. The key: if the prefix is identical, those intermediates are identical —so why reco…
- “The closing point”: Review lens : when every request re-calls the model / recomputes, ask “why isn't this cached?”
The final “The closing point” brings the discussion to “Review lens : when every request re-calls the model / recomputes, ask “why isn't this cached?””. The useful thing to carry forward is knowing which judgments must be revisited when input, scale, or risk changes.
What this lesson wants to share
- Cache = don't recompute: store results by key in a hash table, fetch next time—last lesson's structure starts earning money here
- KV Cache keys on prefix: fixed content at the prompt head, changing bits at the end—hit rate writes the bill
- Semantic cache is fiercer: same-meaning asks skip the model entirely; savings scale linearly with hit rate in high-volume scenes
- Three cache questions: what to store (results worth reuse), where (RAM / disk / near the user), when to invalidate (the hardest)
- Review lens: when every request re-calls the model / recomputes, ask “why isn't this cached?”
I turned one judgment from this article into a small experiment I could run today. Knowing what to observe next is more useful than simply remembering the conclusion.
After reading this, I first looked for the conditions behind the idea instead of copying the method into a project. That order made the later trade-offs much clearer.
When this judgment reaches real work, which constraint should be added first? I am curious which step matters most between reading and the first practical attempt.
No discussion on this article yet.