Why most RAG systems fail in production
The demo works and the production system does not. Almost every time, the failure is in retrieval rather than the model — and it is fixable.
A retrieval-augmented generation prototype takes an afternoon. Load documents, chunk them, embed them, search by cosine similarity, pass the results to a model. It demos beautifully.
Then it meets a real corpus and starts confidently inventing things.
The instinct at that point is to blame the model, and the usual response is to try a bigger one. That almost never helps, because the model is not the component that failed.
The model is not hallucinating, it is guessing
A language model handed weak context does not stop. It produces the most plausible continuation available, and if the retrieved passages are only loosely related to the question, the most plausible continuation is a fluent, confident, wrong answer.
This is the dangerous property of a bad RAG system: its failure output is indistinguishable in tone from its success output. A search engine that finds nothing shows you an empty page. A RAG system that retrieves nothing useful writes you a paragraph.
So the first diagnostic question is never "which model are we using". It is: was the correct passage in the context window at all?
In our experience that answer is no far more often than teams expect.
Vector search alone loses exact terms
Embeddings encode meaning, and that is exactly the problem. Two passages about invoicing policy sit close together in vector space whether or not either one mentions the specific policy reference someone searched for.
Consider what people actually type into an internal knowledge system:
- A part number
- A policy or clause reference
- An error code
- A person's name
- A contract identifier
These are tokens where an exact match matters enormously and semantic similarity is nearly worthless. Vector search will happily return a dozen passages about the general topic and miss the one containing the string.
Hybrid retrieval is the fix
Run both. Dense vector search for semantic similarity, BM25 keyword search for exact terms, then fuse the two ranked lists.
Fuse them with reciprocal rank fusion rather than a weighted score. Weighted scores require tuning a ratio that is wrong for half your queries — semantic weight should be high for a conceptual question and low for a part number lookup, and you do not know which one you got. Reciprocal rank fusion sidesteps this: a document ranked highly by either method surfaces, without either method needing to be correct on its own.
This single change fixes more production RAG complaints than any other we have made.
Chunking is doing more damage than your embedding model
Splitting documents every 500 tokens is the default in every tutorial, and it destroys the thing that makes documents useful: structure.
A clause severed from its heading loses the context that says which section it governs. A table row split from its header row becomes a sequence of meaningless numbers. A definition split across a boundary exists in neither chunk.
Chunk along structural boundaries instead, and carry the heading path with each chunk. A retrieved passage should arrive knowing it is "Refunds → Partial refunds → Eligibility" in revision 4 of a named document. That context costs a handful of tokens and changes the model's output materially.
We have consistently seen larger gains from fixing chunking than from changing embedding models.
Nothing improves without an evaluation set
Most teams building RAG systems evaluate by asking it a few questions and reading the answers. That detects catastrophic breakage and nothing else. It will not tell you that last week's change made retrieval 15% worse.
Build a fixed set of questions with known correct source passages. Thirty is enough to start. Then score two things separately:
- Retrieval — was the correct passage in the retrieved set?
- Generation — did the answer use it and cite it?
Keeping these separate is the entire point. A single blended quality score tells you the system got worse; it does not tell you which half to fix. When retrieval is scored on its own, the answer is usually unambiguous — and usually retrieval.
Make refusal a real, tested behaviour
A system that cannot say "I do not know" will always fabricate, because fabricating is the only remaining option.
Two things make refusal reliable. First, enforce citations structurally: require the model to bind each claim to a retrieved chunk identifier, then strip any claim whose citation does not resolve. An answer that collapses to nothing after stripping becomes an explicit "not found in the available documents".
Second, put deliberately out-of-corpus questions in the evaluation set. If refusal is not tested, it will regress the moment someone adjusts a prompt.
Counter-intuitively, an assistant that occasionally refuses gets trusted more. Users who have seen it decline believe it when it answers.
Permissions belong in retrieval, not in the prompt
If different users may see different documents, filter at retrieval time so restricted documents are never candidates.
Instructing the model not to reveal restricted content is not a control. It is a request, made to a system that is optimising for helpfulness, about material already sitting in its context window. Never put a document in the context of a user who may not read it.
Where to start
If a RAG system is underperforming, work in this order:
- Build an evaluation set and measure retrieval separately from generation
- Add keyword search alongside vector search and fuse the results
- Fix chunking to preserve document structure and headings
- Enforce citations and test refusal explicitly
- Only then consider changing the model
Most projects never get past step three, because by then the problem has usually gone away.
