Free · structured · practical
Search 中文 Start at lesson 1
Courses/Advanced Wrap-Up/Contextual Retrieval: Better RAG
Part 4 · Engineering Patterns for Reliable Agents

Contextual Retrieval: Better RAG

Adding context to chunks before retrieval — Anthropic's upgraded RAG approach

THE QUESTION THIS PAGE ANSWERS

ANSWER FIRST

What is the key idea behind “Contextual Retrieval: Better RAG”?

Adding context to chunks before retrieval — Anthropic's upgraded RAG approach

DECISION RULE

Make the claim earn its place. Use this page as a decision aid, not a definition to memorize. Connect the idea to one real task, one observable result, and one failure that would change your mind.

TRY NEXT

Write one question you could answer with evidence after trying this idea.

WATCH FOR

A conclusion that sounds complete but leaves the key assumption untested.

The Core Problem with Traditional RAG

The traditional RAG workflow: split a document into small Chunks, vectorize each Chunk, retrieve the most similar Chunks when a user asks a question, then feed the Chunks to the LLM to generate an answer. This workflow has one fatal flaw:

Chunks become ambiguous when stripped of context

Once a document is split into Chunks, each Chunk loses its positional information in the original document. Text that was crystal-clear in context may be completely unintelligible on its own.
Typical Example
"The company's Q2 revenue increased by 3% over the previous quarter."
Which company? Which year? What was the Q1 baseline? Is this growth rate good or bad for the industry? All this critical context is lost during chunking. The vector search may find this Chunk, but the Chunk itself carries severely insufficient information.
Full Document
Split into Chunks
Context Lost
Fuzzy Retrieval
The Core Idea of Contextual Retrieval

Use an LLM to prepend a context prefix to each Chunk before Embedding

The idea is elegantly simple: before vectorizing each Chunk, have an LLM read the entire document and generate a brief contextual description as a prefix for each Chunk. This way, every Chunk carries the necessary context when retrieved.
BEFORE -- Bare Chunk
"The company's Q2 revenue increased by 3% over the previous quarter."
Who? When? No way to know.
AFTER -- Chunk with Context
"This chunk is from the company's 2024 Annual Report, specifically the Financial Performance section. The company's Q2 revenue increased by 3% over the previous quarter."
The LLM-generated prefix automatically adds source, time, and section.
Three-Layer Progressive Optimization
LAYER 01
Contextual Embeddings
Vectorize each Chunk after prepending its context prefix. The prefix contains document title, section position, key entities, etc. During retrieval, each Chunk carries its own context for more precise semantic matching.
LAYER 02
Contextual BM25
Traditional BM25 keyword retrieval also benefits from the context prefix. Keywords in the prefix (e.g., company names, years) allow BM25 to match Chunks that would otherwise be missed due to missing context. Vector retrieval + BM25 dual-path recall, complementing each other's blind spots.
LAYER 03
Reranking
After retrieval, use a Reranker model to re-rank candidate Chunks. The Reranker can more precisely judge the relevance between a Chunk and the query, surfacing the most relevant results. All three layers combined yield the best results.
Performance Data

Retrieval Failure Rate Reduction

Contextual Embeddings Only
49%
Contextual Embeddings + BM25 + Reranking
67%
What does a 67% reduction in retrieval failure rate mean? Suppose that previously 30 out of every 100 retrievals failed to find the right Chunk (failure rate 30%). After optimization, the failure rate drops to about 10% — two-thirds of retrieval errors are eliminated. For production systems that rely on RAG, this is a qualitative leap.
Cost Trade-offs

No Free Lunch

Increased pre-processing cost: Each Chunk requires an additional LLM call to generate its context prefix. For large-scale document libraries, this pre-processing cost is significant.
Prompt Caching can reduce cost: Different Chunks from the same document share the same document-level context. Using Prompt Caching avoids repeatedly sending the entire document.
Best suited for high-accuracy scenarios: If your RAG system demands very high accuracy (e.g., legal document retrieval, medical Q&A, financial compliance queries), the extra pre-processing cost is worth it. For high-tolerance scenarios (e.g., casual recommendations), it may not be cost-effective.
RAG is not just about chunking plus vector retrieval — every Chunk must carry its own context. The core insight of Contextual Retrieval: the quality bottleneck in retrieval lies in the informational completeness of the Chunk itself; switching to a stronger embedding model helps only marginally. Restoring the lost context to each Chunk can reduce retrieval failure by two-thirds.

Why “The Core Problem with Traditional RAG” can find relevant content

“The traditional RAG workflow: split a document into small Chunks, vectorize each Chunk, retrieve the most similar Chunks when a user asks a question, then feed the Chunks to the LL…” moves retrieval beyond storing material: the real question is how to find what is relevant. That decision shapes the input quality of RAG, recommendation, and image-search systems.

Similarity is not the answer

In the flow described by “The traditional RAG workflow: split a document into small Chunks, vectorize each Chunk, retrieve the most similar Chunks when a user asks a question, then feed the Chunks to the LL…”, embeddings place items in a comparable semantic space and a neighbor index narrows the search. The final answer still depends on whether the retrieved chunks cover the question, whether the distance metric fits, and whether the evidence is current.

Separate findable from relevant

Turn “The traditional RAG workflow: split a document into small Chunks, vectorize each Chunk, retrieve the most similar Chunks when a user asks a question, then feed the Chunks to the LL…” into a small test: prepare queries with known answers, record relevance, misses, and distractors, then decide whether chunking, the index, or reranking needs to change.

From “The Core Problem with Traditional RAG” to “The Core Idea of Contextual Retrieval”

“The Core Problem with Traditional RAG” grounds the problem in “The traditional RAG workflow: split a document into small Chunks, vectorize each Chunk, retrieve the most similar Chunks when a user asks a question, then feed the Chunks to the LLM to generate an answer. This…”. “The Core Idea of Contextual Retrieval” then moves it toward “Use an LLM to prepend a context prefix to each Chunk before Embedding The idea is elegantly simple: before vectorizing each Chunk, have an LLM read the entire document and generate a brief contextual descriptio…”. Together, they show that the lesson is not just a conclusion to remember, but a claim with conditions.

Carry the judgment into the next situation

The same logic applies to retrieval: define what counts as relevant, check whether recall covers the question, and then inspect whether ranking, chunking, or freshness pushed useful evidence out.

  • “The Core Problem with Traditional RAG”: The traditional RAG workflow: split a document into small Chunks, vectorize each Chunk, retrieve the most similar Chunks when a user asks a question, then feed the Chunks to the LLM to generate an answer. This…
  • “The Core Idea of Contextual Retrieval”: Use an LLM to prepend a context prefix to each Chunk before Embedding The idea is elegantly simple: before vectorizing each Chunk, have an LLM read the entire document and generate a brief contextual descriptio…
  • “The closing point”: The traditional RAG workflow: split a document into small Chunks, vectorize each Chunk, retrieve the most similar Chunks when a user asks a question, then feed the Chunks to the LLM to generate an answer. This…

The final “The closing point” brings the discussion to “The traditional RAG workflow: split a document into small Chunks, vectorize each Chunk, retrieve the most similar Chunks when a user asks a question, then feed the Chunks to the LLM to generate an answer. This…”. The useful thing to carry forward is knowing which judgments must be revisited when input, scale, or risk changes.

Mark as learned Your reading progress updates automatically
← PreviousNext →

Keep reading

The next useful article in the thread.

Advanced Wrap-Up · RecapAdvanced OverviewDesign patterns, tools, evaluation, long-running, security — all connected in one diagramAdvanced Wrap-Up · FinaleDo the Simplest Thing That WorksAnthropic's core engineering philosophy: "Do the simplest thing that works"Long-Running Agents · Deep DiveSession ≠ Context WindowSession logs are persistent event streams; the context window is temporary working memory — they must be separatedThe Art of Tool Design · Deep DiveThink Tool: Making AI Think Before ActingGive the Agent a pause-to-think space in complex tool chains — τ-bench performance up 54%

ARTICLE DISCUSSION

Leave one useful thought here.

Keep the idea that clicked, the question that stayed open, or a small note for the next learner.

↗
Discussing Contextual Retrieval: Better RAG Engineering Patterns for Reliable Agents
3discussionsArticle discussion · synced with the Circle
View in the learning circle
AM
Asha MorganContent editor
INSIGHTField note

I turned one judgment from this article into a small experiment I could run today. Knowing what to observe next is more useful than simply remembering the conclusion.

ARTICLE DISCUSSION7 helpful
LH
Lin HarperIndie developer
INSIGHTInsight

After reading this, I first looked for the conditions behind the idea instead of copying the method into a project. That order made the later trade-offs much clearer.

ARTICLE DISCUSSION5 helpful
KM
Kiki MooreProduct operations
QUESTIONQuestion

When this judgment reaches real work, which constraint should be added first? I am curious which step matters most between reading and the first practical attempt.

ARTICLE DISCUSSION4 helpful