As we build increasingly sophisticated RAG (Retrieval-Augmented Generation) systems, we encounter a persistent challenge: ensuring the AI stays true to its source material. It’s a common misconception that simply instructing a Large Language Model (LLM) to “answer based only on the provided context” is sufficient. In reality, preventing hallucinations and ensuring high-fidelity answers requires robust engineering mechanisms, not just prompt engineering.
In this post, we’ll explore why simple prompting falls short and detail the specific mechanisms we’ve implemented—like granular verification and source narrowing—to provide deeper, more reliable answers.
If you are building RAG systems and care about answer fidelity, this is worth 15 minutes.
The Challenge: Why “Don’t Hallucinate” Isn’t Enough
The most intuitive approach to RAG is simple: retrieve relevant documents, feed them to the LLM, and add a system instruction like:
“Answer the user’s question using only the provided context. Do not use outside knowledge. If the answer isn’t in the context, say you don’t know.”
While this helps, it is far from failsafe. LLMs are trained to be helpful and creative completion engines. When faced with a subtle gap in the provided context, they often “bridge the gap” with plausible-sounding but unverified information from their pre-training data. This “hallucination” is often subtle—a right answer, but for the wrong version of a product, or a conflation of two different documents.
Furthermore, when we inject 10, 20, or 30 document chunks into the context window to maximize coverage, we introduce noise. The model might latch onto a semantically similar but irrelevant chunk, leading to an answer that is “grounded” in the wrong source.
Our Approach: Trust Through Verification
To solve this, we moved beyond passive prompting to active verification. We treat the LLM’s initial answer not as the final product, but as a draft that must undergo rigorous fact-checking before reaching the user.
Our system implements a multi-stage fidelity pipeline designed to catch hallucinations at a granular level.
1. Granular Verification: The Paragraph Test
One of our key insights was that hallucinations are often localized. An answer might be 90% correct, with just one sentence drifting into fabrication. To catch this, we implemented per-paragraph keyword verification.
Instead of checking the answer as a vague whole, our FidelityService breaks the generated answer into individual paragraphs. For each paragraph, we:
- Extract Significant Keywords: We ask the model to identify the key entities and claims (topics, specific values, names) in that specific paragraph.
- Verify Presence: We programmatically check if these keywords actually exist in the source documents.
- Strict Thresholding: We enforce a configurable threshold (e.g., 35% of keywords must be explicitly found). If any paragraph fails this test—even if the rest of the answer is perfect—flag it for a redo.
This granular approach prevents “partial hallucinations” from slipping through. An answer cannot ride on the coattails of a mostly correct summary; every claim must earn its keep.
2. Source Narrowing: Providing Better Context
A major cause of hallucination is “context flooding”—giving the model too much information. When a user asks a specific question, they don’t need 20 loose chunks of text; they often need one complete, coherent document.
We addressed this with a Two-Phase Source Narrowing strategy:
- Phase 1 (Citation Check): When the model generates an initial answer, it cites specific documents. We verify these citations first. If the keywords from the answer are largely found in the cited docs, we know the model is on the right track.
- Phase 2 (Context Refinement): If the verification fails or needs a redo, we don’t just ask the model to “try again” with the same overwhelmed context. Instead, we narrow the source scope:
- If the model cited 1-2 specific documents, we retrieve the full text of those documents (replacing the fragmented chunks) to give the model complete context.
- We remove irrelevant chunks that might have distracted the model.
- We essentially say: “You identified Document A and B as relevant. Here is the full text of A and B. Now answer the question again strictly using these.”
By narrowing the scope to the most probable sources, we remove the noise that causes hallucinations.
Conclusion
Building a trustworthy RAG system isn’t about finding the perfect prompt; it’s about building a verification loop. By implementing granular paragraph-level checks and intelligently narrowing source context based on initial citations, we can move from “hoping” the model doesn’t hallucinate to proving it hasn’t.
This engineering-first approach allows us to trust the answers our system provides, knowing they are backed by specific, verified evidence.
Further Reading
- IBM Think – AI Hallucinations: What They Are and How to Reduce Them
- Microsoft Azure AI Foundry Blog – Best Practices for Mitigating Hallucinations in Large Language Models (LLMs)
