From File Changes to Hybrid Ranking
Tracing the memory recall pipeline: FTS, vector retrieval, time decay, and MMR re-ranking
THE QUESTION THIS PAGE ANSWERS
ANSWER FIRSTWhat is the key idea behind “From File Changes to Hybrid Ranking”?
Tracing the memory recall pipeline: FTS, vector retrieval, time decay, and MMR re-ranking
Make the claim earn its place. Use this page as a decision aid, not a definition to memorize. Connect the idea to one real task, one observable result, and one failure that would change your mind.
Write one question you could answer with evidence after trying this idea.
A conclusion that sounds complete but leaves the key assumption untested.
Sync Before Query
MemoryFileWatcher accumulates changed Markdown paths. The backend re-indexes new or modified files at the start of each search, and deletes stale chunks for removed files.
BM25 Candidates
First run standard FTS, then supplement with global and workspace source queries to reduce crowding-out caused by too many sessions.
Optional KNN
Embeds the query only when sqlite-vec and the provider are available. Embedding errors are logged as warnings; None is passed to continue FTS-only.
Normalize & Merge
BM25 scores and vector L2 distances are normalized separately. Dual-path hits are merged by weight, while ensuring results are not below the chunk's FTS score.
Time & Source
Sessions decay exponentially with a half-life; global and workspace sources are treated as evergreen. Then multiply by source weight and a moderate access boost.
Optional MMR
When enabled, performs greedy re-ranking by relevance and Jaccard diversity of snippets. Finally truncates to max_results.
Embedding Failure
The vector path stops, but FTS results still enter hybrid_search_merge. The page or caller does not need to treat an embedding failure as a total search failure.
MMR Default State
MmrConfig::default() sets enabled: false and lambda: 0.7. The 0.7 value only takes effect when MMR is explicitly enabled.
pub async fn hybrid_search(
index: &MemoryIndex,
embedding_provider: Option<&dyn EmbeddingProvider>,
query: &str,
config: &MemorySearchConfig,
) -> Result<Vec<SearchResult>, Box<dyn std::error::Error>> {
let candidate_limit = config.max_results * 3;
let mut fts_results =
index.search_fts(query, candidate_limit).unwrap_or_default();
/* source for supplementing evergreen FTS candidates is here */
let vec_available = index.vec_available();
let query_embedding = if vec_available {
if let Some(provider) = embedding_provider {
match provider.embed_batch(&[query]).await {
Ok(embeddings) if !embeddings.is_empty() =>
Some(embeddings.into_iter().next().unwrap()),
Ok(_) => None,
Err(e) => {
tracing::warn!(error = %e,
"embedding query failed, falling back to FTS-only");
None
}
}
} else { None }
} else { None };
hybrid_search_merge(index, fts_results, query_embedding.as_deref(), config)
}crates/codegen/xai-grok-memory/src/backend.rs: search() — executes watcher sync and query
crates/codegen/xai-grok-memory/src/watcher.rs: MemoryFileWatcher
crates/codegen/xai-grok-memory/src/mmr.rs: mmr_rerank
crates/codegen/xai-grok-config-types/src/memory.rs: MmrConfig defaults
grok-build-main, verified on 2026-07-17. Code blocks retain real functions and branches; the only omitted section is explained by a comment; the flow diagram is explicitly labeled as a pedagogical diagram.Trace a Degraded Query
Assume the watcher detects a modified file, query embedding then fails, and MMR stays at its default configuration. Write out — in order — index update, candidate generation, weighted ranking, and final truncation, and mark the two steps that did not occur.
Why “Core Diagram · Complete Retrieval Path” can find relevant content
“MemoryFileWatcher accumulates changed Markdown paths.” moves retrieval beyond storing material: the real question is how to find what is relevant. That decision shapes the input quality of RAG, recommendation, and image-search systems.
Similarity is not the answer
In the flow described by “First run standard FTS, then supplement with global and workspace source queries to reduce crowding-out caused by too many sessions”, embeddings place items in a comparable semantic space and a neighbor index narrows the search. The final answer still depends on whether the retrieved chunks cover the question, whether the distance metric fits, and whether the evidence is current.
Separate findable from relevant
Turn “Assume the watcher detects a modified file, query embedding then fails, and MMR stays at its default configuration.” into a small test: prepare queries with known answers, record relevance, misses, and distractors, then decide whether chunking, the index, or reranking needs to change.
From “Core Diagram · Complete Retrieval Path” to “Sync Before Query”
“Core Diagram · Complete Retrieval Path” grounds the problem in “Watcher Dirty Paths create · modify · remove sync-on-search reindex_file / delete_path User Query query FTS5 BM25 Always available · keyword candidates Embedding + KNN Uses sqlite-vec when available embedding f…”. “Sync Before Query” then moves it toward “MemoryFileWatcher accumulates changed Markdown paths. The backend re-indexes new or modified files at the start of each search, and deletes stale chunks for removed files”. Together, they show that the lesson is not just a conclusion to remember, but a claim with conditions.
Carry the judgment into the next situation
The same logic applies to retrieval: define what counts as relevant, check whether recall covers the question, and then inspect whether ranking, chunking, or freshness pushed useful evidence out.
- “Core Diagram · Complete Retrieval Path”: Watcher Dirty Paths create · modify · remove sync-on-search reindex_file / delete_path User Query query FTS5 BM25 Always available · keyword candidates Embedding + KNN Uses sqlite-vec when available embedding f…
- “Sync Before Query”: MemoryFileWatcher accumulates changed Markdown paths. The backend re-indexes new or modified files at the start of each search, and deletes stale chunks for removed files
- “The closing point”: MmrConfig::default() sets enabled: false and lambda: 0.7 . The 0.7 value only takes effect when MMR is explicitly enabled
The final “The closing point” brings the discussion to “MmrConfig::default() sets enabled: false and lambda: 0.7 . The 0.7 value only takes effect when MMR is explicitly enabled”. The useful thing to carry forward is knowing which judgments must be revisited when input, scale, or risk changes.
I turned one judgment from this article into a small experiment I could run today. Knowing what to observe next is more useful than simply remembering the conclusion.
After reading this, I first looked for the conditions behind the idea instead of copying the method into a project. That order made the later trade-offs much clearer.
When this judgment reaches real work, which constraint should be added first? I am curious which step matters most between reading and the first practical attempt.
No discussion on this article yet.