Retrieval is the product
In every RAG system I have shipped, the model was rarely the bottleneck. Retrieval was. If the right passage is not in the context window, no amount of prompting saves the answer.
Chunking with intent
Fixed-size chunking is a convenience, not a design. Splitting on structure — headings, function boundaries, table rows — consistently beat token-count splitting in every evaluation I ran.
Hybrid retrieval, then rerank
dense = vector_store.search(query, k=50)
sparse = bm25.search(query, k=50)
candidates = reciprocal_rank_fusion(dense, sparse)
return reranker.rank(query, candidates)[:8]Dense retrieval finds paraphrases. Sparse retrieval finds identifiers, error codes and version numbers. You need both.
Evaluate like an engineer
Build a small golden set of real questions with known answers before you tune anything. Without it, every change is a vibe. With it, you can say that recall@8 went from 0.61 to 0.88 and mean an actual thing.