Why retrieval-augmented generation still needs hybrid search
Retrieval-augmented generation puts a language model on top of a search step: retrieve the most relevant passages, then generate an answer from them. The model does what it always does — write fluent, confident text. Whether that text is correct depends almost entirely on whether the right passages were retrieved in the first place. A wrong retrieval doesn't produce an error message; it produces a wrong answer that reads exactly as confidently as a right one.
Where internal knowledge bases break retrieval
The hardest RAG targets aren't public documents — they're the manuals, tickets, contracts and ERP and CRM records a company already has, full of internal abbreviations, part numbers and proper nouns that never appear in a model's training data. As explained in how vector embeddings work, embeddings match on meaning, not exact strings — which is precisely why an internal ticket code or a rare SKU can get lost in a pile of passages that are merely topically similar, without ever being flagged as a miss.
One retrieval method isn't enough
Why hybrid search beats either alone makes the case for product search; the same trade-off applies one layer up in a RAG pipeline, with higher stakes. Whatever the retriever fails to surface, the model never sees — and no amount of prompt engineering recovers a passage that was never retrieved. Keyword matching and semantic matching fail in different, complementary places, so a RAG retriever that only runs one of them inherits that method's blind spot directly into the model's context window.
Reranking is where the two lists actually merge
Running both retrievers only produces two ranked lists; something still has to decide the final order the model sees. A cross-encoder reranker scores query–passage pairs directly, rather than comparing precomputed vectors, which is slower and more expensive per query but meaningfully more accurate at picking the few passages that actually belong in a limited context window. For a RAG system, that's usually worth it: the cost of reranking twenty candidate passages is small next to the cost of the model generating a wrong answer from the top five.
The practical takeaway
Model choice gets most of the attention in RAG projects; retrieval quality is usually the bigger lever. Before tuning prompts or swapping models, check what the retriever actually returns for the queries that matter — exact identifiers, paraphrased questions, and anything that requires connecting two separate passages. If any of those routinely come back empty or wrong, that's the fix to make first.