Semantic Layer: Double Distillation
Lost-in-the-middle: the more you stuff in, the less it holds onto what matters. Dynamic Few-Shot cuts 4000 to 500; LLMLingua-2 compresses 5–20×
THE QUESTION THIS PAGE ANSWERS
ANSWER FIRSTWhat is the key idea behind “Semantic Layer: Double Distillation”?
Lost-in-the-middle: the more you stuff in, the less it holds onto what matters. Dynamic Few-Shot cuts 4000 to 500; LLMLingua-2 compresses 5–20×
Read cost as a shape, not a single number. Break a request into input, output, retries, tools, and waiting time. The shape of usage usually tells you which design choice is expensive and where a smaller change can help.
Measure one real request before you optimize an imagined average.
A cheaper call that quietly creates more retries, latency, or review work.
First: expensive and slow. Transformer self-attention is O(N²): double the prompt length, and compute grows 4×. Longer prompts mean longer Prefill and higher time to first token—users lose patience before they see the first character.
Second: quality can get worse. When signal drowns in filler, you get the lost-in-the-middle effect. Like a person reading a long article: they focus at the start (what's this about?), pay attention at the end (here's the conclusion), and skim the big middle blob—eyes pass, brain doesn't. If your carefully picked references land in the middle, the model may never really read them.
The color strip simulates attention strength across Prompt positions (green = strong, gray = weak). Drag the length and watch the middle sag.
Take Text-to-SQL: to cover every business case, some people hard-code 20 SQL examples into the Prompt—4,000+ Tokens total. Every user question, the model “reviews” all 4,000 first: burns Tokens and runs slow. Better approach:
Store the 20 examples in a vector database.
When the user asks “last month's sales,” use semantic retrieval to pull only the Top-3 finance-related examples.
Final Prompt drops from 4,000 Tokens to 500.
(no irrelevant-example noise)
Financial research, meeting notes—docs packed with “correct filler”: disclaimers, repeated background, spoken padding. Feed that to the AI and you're paying PhD rates to read spam.
Fix: after RAG retrieval and before inference, insert an LLMLingua-2 middleware. It doesn't chop words blindly: BERT bidirectional attention sees both sides of context, pinpoints core semantics (entities, numbers, key verbs), and strips redundant noise. Compress 5–20×; Prefill that took 1 second drops to 50 ms—high-concurrency throughput jumps a full order of magnitude.
| Distillation | Target | Method | Typical gain |
|---|---|---|---|
| First pass | Few-Shot examples | Vector retrieval picks Top-K dynamically | 4,000 → 500 Tokens |
| Second pass | Retrieved documents | LLMLingua-2 semantic compression | 5–20× compression; Prefill 1s → 50ms |
Double the prompt = 4× the compute: O(N²) is why long context is both expensive and slow.
Lost-in-the-middle: put critical info at the start or end; leave the middle for content you wouldn't mind losing.
Don't hard-code Few-Shot—store in a vector DB and retrieve by question: save 87.5% and get more accuracy.
Run long docs through LLMLingua-2 before inference: high-density Prompt for high-quality Attention—don't lose users while they wait.
Source: Adapted from the author's internal team share “AI Token Cost Engineering Strategy” hands-on section “02｜Semantic Layer.” For lost-in-the-middle, see Chroma's Context Rot research; compression benchmarks at LLMLingua. Review context-window basics in Harness Core · Context Window.
The complete interaction cost of “Two sins of dump-style context”
“First: expensive and slow.” is a reminder that AI cost is not one price multiplied by one call. Input, output, retries, tools, waiting time, and human cleanup together decide what a task really costs.
Find what the bill repeats
The key variables behind “Second: quality can get worse.” are usually repeated context, oversized output, retries after failure, and calls that do not produce useful progress. Removing wasted Tokens can reduce cost, latency, and concurrency pressure at the same time.
A cheaper call can make the whole workflow more expensive
Start with “Run long docs through LLMLingua-2 before inference: high-density Prompt for high-quality Attention—don't lose users while they wait” and keep a small table for input, output, retries, tools, and human review. Compare quality before and after optimizing instead of looking at one price in isolation.
From “Two sins of dump-style context” to “Interactive Demo · How attention gets diluted”
“Two sins of dump-style context” grounds the problem in “First: expensive and slow. Transformer self-attention is O(N²): double the prompt length, and compute grows 4×. Longer prompts mean longer Prefill and higher time to first token—users lose patience before they…”. “Interactive Demo · How attention gets diluted” then moves it toward “The color strip simulates attention strength across Prompt positions (green = strong, gray = weak). Drag the length and watch the middle sag”. Together, they show that the lesson is not just a conclusion to remember, but a claim with conditions.
Carry the judgment into the next situation
When analyzing cost, map the complete interaction first, then find repeated input, wasted output, and retries. A cheap individual call does not make the whole task cheap.
- “Two sins of dump-style context”: First: expensive and slow. Transformer self-attention is O(N²): double the prompt length, and compute grows 4×. Longer prompts mean longer Prefill and higher time to first token—users lose patience before they…
- “Interactive Demo · How attention gets diluted”: The color strip simulates attention strength across Prompt positions (green = strong, gray = weak). Drag the length and watch the middle sag
- “The closing point”: Financial research, meeting notes—docs packed with “correct filler”: disclaimers, repeated background, spoken padding. Feed that to the AI and you're paying PhD rates to read spam
The final “The closing point” brings the discussion to “Financial research, meeting notes—docs packed with “correct filler”: disclaimers, repeated background, spoken padding. Feed that to the AI and you're paying PhD rates to read spam”. The useful thing to carry forward is knowing which judgments must be revisited when input, scale, or risk changes.
I turned one judgment from this article into a small experiment I could run today. Knowing what to observe next is more useful than simply remembering the conclusion.
After reading this, I first looked for the conditions behind the idea instead of copying the method into a project. That order made the later trade-offs much clearer.
When this judgment reaches real work, which constraint should be added first? I am curious which step matters most between reading and the first practical attempt.
No discussion on this article yet.