Most “hallucination” complaints in retrieval-augmented systems are retrieval failures. If the wrong chunk, an unauthorized chunk, or a stale chunk reaches the model, fluent answers will still be wrong. Fix evaluation around retrieval before you swap models.
Citation checks
Require answers to point at retrieved passages. Spot-check whether the cited text actually supports the claim. If citations are missing or ornamental, treat the run as a fail even when the prose sounds right.
Track citation precision over time. A model upgrade that drops citation honesty is a regression.
Golden questions
Build a small set of real questions with known-good document IDs. Score whether the retriever returns those docs in the top-k before you score the generated answer.
Include negatives: questions that should return nothing, or should refuse when docs are insufficient.
Permission boundaries
Filter retrieval by the user’s ACL before ranking. Never retrieve then redact in the prompt and hope the model complies. Cross-tenant or cross-team leakage is a product incident, not an accuracy issue.
Test with users who should see nothing for a given corpus. Empty results are the correct answer.
Stale docs
Index freshness, document owners, and deprecation flags belong in the pipeline. A perfect embedding of last year’s policy is still a wrong answer.
When source systems change, re-ingest on a schedule you can explain—and show “as of” dates in the UI when it matters.
Order of work
Instrument retrieval metrics first. Then tune chunking, embeddings, and hybrid search. Only then compare generators on the same retrieved context.
If retrieval is wrong, a better model mostly writes a better wrong answer.