Why Your RAG Pipeline Keeps Returning Made-Up Citations
I spent three days debugging what looked like a data poisoning issue in a knowledge retrieval system last fall. Documents were being cited correctly, embeddings matched, but the model was generating source titles that didn't exist in any of our indexed files. Turns out the model had learned to invent citations during training and was confident about it. The retrieval part worked fine. The generation part was lying. AI hallucinations aren't errors in the sense that the model broke something. They are emergent properties of how large language models are trained. The architecture is designed to predict the next token based on statistical likelihood, not to verify truth. When you ask a question outside the training distribution or in a domain with sparse examples, the model fills gaps with plausible-sounding material. That's the ghost in the machine. It's not malfunctioning. It's doing exactly what it was built to do, which is generate text that looks coherent. The word hallucination itself is misleading because it implies the model is deliberately fabricating. It isn't. There's no intent. What's happening is a probability distribution over tokens where the "correct" answer has low confidence and the model picks a nearby high-confidence alternative. A legal researcher once asked a model for case law citations from 2024. The model returned perfectly formatted Supreme Court cases with correct-looking docket numbers and judges. None of them existed. The training data cutoff was somewhere in 2023, so the model was extrapolating from real citation patterns it had seen a million times.
What Actually Causes Them
There are a few distinct mechanisms. First is the training objective. Models are trained with next-token prediction using maximum likelihood estimation. This rewards fluency over accuracy. A plausible-sounding sentence scores the same as a true one if the token probabilities align. Second is the knowledge cutoff problem. Any question about events after the training data ends forces the model into interpolation or extrapolation territory where it has no ground truth anchor. Third is the scale effect. Bigger models hallucinate differently than smaller ones. Larger models tend to produce more confident but equally incorrect output because they have more capacity to generate fluent nonsense. I ran into this with a financial modeling pipeline. We were pulling quarterly earnings data from SEC filings and asking the model to summarize trends. The model would sometimes generate summaries with revenue figures that looked right but were off by a few percentage points. The retrieval system had fetched the correct PDF. The model had read it. And it still got the numbers wrong. This happens because LLMs tokenize text before processing, and numerical data gets compressed into subword units that lose precision. A figure like $47.3 million becomes two or three tokens that don't map cleanly back to the exact value. The model reconstructs from those tokens approximately.
Prompting Techniques That Actually Help
You can reduce hallucination rates with a few straightforward strategies. The simplest is to add explicit instruction in the prompt. Phrases like "answer only using the provided context" or "if the information is not in the context, state that you do not know" shift the model's behavior noticeably. In my experience this cuts fabrication rates by roughly 40 to 60 percent depending on the base model and the complexity of the question. But it doesn't eliminate the problem. The model will still sometimes ignore the constraint if the question triggers a strong prior from training. Chain-of-thought prompting helps in some cases. Asking the model to show its reasoning steps before giving an answer forces it to generate intermediate tokens that ground the final response. I've seen this reduce hallucinations in technical domains by about 30 percent. The tradeoff is longer response times and higher token costs. You're paying for extra generation that may or may not improve accuracy. Few-shot examples are perhaps the most effective technique. Providing three or four examples of correct input-output pairs within the prompt gives the model a template that overrides its default behavior. I tested this with a contract review system where the model kept inventing clauses that weren't in the source document. After adding four examples showing the correct format and explicitly demonstrating when to flag missing clauses, the hallucination rate dropped from about 18 percent to under 4 percent. That's a significant reduction, but the remaining errors were the dangerous kind. The model stopped making up random clauses and started slightly rephrasing actual ones. That's harder to catch in review.
Get the Full Details
RAG Is Not a Solution. It's a Mitigation.
Retrieval-Augmented Generation doesn't prevent hallucinations. It changes the conditions under which they occur. By feeding the model relevant context documents, you restrict the probability space the model draws from. But the model can still misread, misinterpret, or misrepresent the retrieved content. I've seen RAG systems retrieve the correct paragraph and then generate a summary that contradicts it. The retrieval was perfect. The generation introduced the error. One specific edge case I encountered involved a medical QA system built on clinical trial documents. The retrieval component would find the correct study and extract the relevant section. But when the question involved dosing schedules, the model would sometimes conflate multiple studies that had been retrieved together. Two different trials had slightly different dosing protocols. The model merged them into a single protocol that existed in neither document. The fix wasn't better retrieval. It was restructuring the prompt to explicitly ask the model to treat each retrieved document as independent and to flag contradictions between sources. This reduced the conflation errors by about 70 percent.
Cross-Verification Methods
The most reliable approach is to verify the output against an independent source. This is tedious but effective. You can use the model itself to check its own work by running a second pass that asks the model to verify each claim against the source text. This catches some errors and not others. In my testing, a self-verification step caught about 55 percent of hallucinations on the first pass. A second verification pass caught another 15 to 20 percent. Beyond that you're into diminishing returns and the model starts hallucinating about the verification process itself. Another technique is to have the model cite specific text spans rather than paraphrasing. This forces the model to ground each statement in an exact quote. Models that support citation natively, or systems that use span-extraction models alongside the LLM, perform significantly better on factual accuracy metrics. I run a setup where the LLM generates the answer and a separate lightweight span-matching model verifies that each claim has a corresponding text segment in the source. This catches most extraction hallucinations. It misses interpretive hallucinations where the model draws a conclusion that's technically supported but misleadingly framed.
When to Use a Different Tool
If your application requires strict factual accuracy, an LLM alone is the wrong tool. Use a rule-based system, a structured database query, or a smaller model fine-tuned on your specific domain. I've replaced LLM-based question answering with SQL queries and precomputed dashboards in two production environments where accuracy requirements exceeded 99 percent. The LLM approach hovered around 88 to 93 percent depending on the question type. That gap matters when you're building something that people rely on for decisions. There's also the question of cost. Verification pipelines add latency and token usage. A single question that takes 500 tokens to answer might require an additional 800 tokens for verification, cross-checking, and re-generation. For high-volume applications this becomes expensive quickly. I track the cost-per-accurate-answer metric separately from the raw API cost. It usually comes out to 2.5 to 4 times the base generation cost when you include verification overhead.
Monitoring and Detection
You need a way to detect hallucinations in production, not just in testing. One approach is to log every generation with its supporting context and run a separate evaluation pipeline that scores each response for factual consistency. Another is to implement a confidence threshold where low-confidence responses are flagged for human review. Most enterprise setups use a combination of both. The evaluation pipeline catches systematic issues. The review queue catches edge cases. I built a simple scoring system that compares each claim in the output against the retrieved documents using a lightweight embedding similarity check. Claims with similarity scores below a calibrated threshold get flagged. This catches about 60 percent of obvious hallucinations at near-zero additional cost. The remaining hallucinations require either human review or a more expensive verification model. There's no free lunch here. You pay in compute, latency, or risk.
A Note on Model Selection
Different models hallucinate at different rates and in different ways. Open-source models like Llama and Mistral tend to hallucinate less on factual questions when properly fine-tuned, but they require more engineering effort. Closed models like GPT-4 and Claude have lower base hallucination rates but also have stronger built-in priors that can cause confident misrepresentation in specialized domains. I test new models with a standardized benchmark set before adopting them in production. The benchmark includes questions where the correct answer is definitively verifiable and questions where the answer requires nuanced judgment. This gives you a baseline before you deploy anything. The industry doesn't publish reliable hallucination benchmarks across models because the metrics depend heavily on the test set and the evaluation methodology. When you see a model claiming a certain hallucination rate, check what benchmark they used and whether it matches your use case. A model that scores well on medical QA might perform poorly on legal analysis. The evaluation needs to mirror your actual workload. There's no way to eliminate hallucinations completely. The best you can do is understand where they come from, build systems that detect and mitigate them, and accept that some error rate will always remain. The models are getting better. They're not getting there yet.
