Is a bigger model the fix for a hallucinating RAG assistant?

Rarely. The generator conditions on retrieved context; if retrieval is wrong, a larger model just produces a more convincing wrong answer. Fix retrieval first.

What should you measure before touching the model?

Retrieval recall and precision on a labeled query set, scored independently of generation, so you know whether the right passage even reaches the model.

Does dense retrieval automatically beat keyword search?

No. Learned dense retrieval can outperform BM25, but only when the embeddings match your corpus; off-domain vectors quietly miss the relevant passage.

Your RAG Assistant Slipped Because of Retrieval, Not the

A post-mortem of the RAG assistant that slipped: the generator was fine, the index was rotten.

Verdict: use RAG, but retrieval — not the model — decides whether it ships. A generator conditions on whatever passages the retriever returns, so a bigger model just writes fluent wrong answers over bad context. Fix chunking, embeddings, and index freshness, and measure retrieval recall before you touch the model.

The Weights Desk · 4 min read 2026-07-10 2026-07-10T09:00:00Z

In RAG, the model answers from retrieved passages — bad retrieval yields confident, fluent wrong answers.
Retrieval is an independent lever: chunking, embedding choice, and index freshness move quality more than model size.
Measure retrieval recall before generation — you cannot fix what you do not score.
Dense retrieval beats BM25 only when the embeddings actually fit your domain; off-domain vectors miss silently.
Bigger context windows are not a fix; stuffing more chunks buries the right passage and raises cost.

Verdict: retrieval quality was the killer, not the model

You bought the bigger model. You rewrote the system prompt twice. The assistant still quotes a refund window that does not exist. The reflex is to blame the generator — and the reflex is almost always wrong. In a retrieval-augmented system the model answers from whatever the retriever hands it, so a smarter model over the wrong passage just writes a more fluent wrong answer. Verdict: use RAG, but treat retrieval as the product and the model as a commodity. The paper that named the pattern — Lewis et al., 'Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks' (arXiv:2005.11401, s1) — drew the line explicitly: a parametric generator conditioned on a separate, non-parametric index. Quality lives in the index.

What actually slipped

The slip follows one script. Chunk boundaries split a policy across two fragments, so the answer is never retrievable whole. The embedding model was trained on general web text and quietly misses your domain's vocabulary. The index is weeks stale, so the retriever confidently returns last quarter's number. Top-k was set to a value nobody tuned. None of that is a model defect. The RAG paper (s1) is built around exactly this separation — the retriever selects passages, the generator writes over them — which is precisely why the retriever is where you debug. Dense retrieval helps, but not automatically: the DPR work (s2) showed learned dense retrieval can beat classical BM25, yet only when the embeddings actually fit the corpus you are searching. Off-domain vectors miss silently, and the silence is the dangerous part.

What would have killed it — and what would have shipped it

Here is what did not work: enlarging the context window. Stuffing fifty chunks in raised token cost, slowed responses, and buried the one passage that mattered — recall went up, answer quality did not. Re-ranking helped, but how much it buys you alone is domain-dependent and, honestly, untested at the scale most teams run. What ships the assistant is boring: build a labeled query set, measure retrieval recall and precision independently of generation, fix chunking so atomic facts stay intact, pick embeddings that match your domain, and keep the index fresh — the original framework treats that index as swappable memory (s1) precisely so you can update knowledge without retraining anything. Do that and RAG ships. Skip it and no model on the market saves you. Use it — but score retrieval first.

Is a bigger model the fix for a hallucinating RAG assistant?: Rarely. The generator conditions on retrieved context; if retrieval is wrong, a larger model just produces a more convincing wrong answer. Fix retrieval first.
What should you measure before touching the model?: Retrieval recall and precision on a labeled query set, scored independently of generation, so you know whether the right passage even reaches the model.
Does dense retrieval automatically beat keyword search?: No. Learned dense retrieval can outperform BM25, but only when the embeddings match your corpus; off-domain vectors quietly miss the relevant passage.