What is it?
A hallucination is a model output that is fluent and confident but wrong: an invented statistic, a book that does not exist, a legal case that was never decided, a function that is not in the library. Two useful distinctions:
- Factual errors: the output contradicts reality.
- Unfaithfulness: the output contradicts or goes beyond the source material you provided (for example, a summary that adds claims the document never made).
The word is a metaphor; the model is not perceiving things that are not there, it is generating text that fits a pattern.
Why does it matter?
Fluent text is persuasive, so errors can pass unnoticed. Real-world consequences include fabricated citations in documents, incorrect customer advice, flawed code that looks right, and decisions made on invented figures. In regulated or high-stakes settings, unchecked output can create legal and safety risk.
How does it work?
A language model is trained to produce likely continuations of text, not to consult a verified fact store (What is a Large Language Model (LLM)?). Several things raise the chance of fabrication:
- No relevant knowledge. For obscure topics or events after its training data, the model still produces something plausible rather than stopping.
- Pressure to answer. A prompt that demands a specific answer (a name, a citation, a number) invites invention.
- Ambiguity or false premises in the question ("Why did company X win the 2019 award?" when it did not).
- Weak or conflicting context in the prompt.
- Long chains of generation, where an early mistake gets built upon.
Models can be well-calibrated in some settings and badly overconfident in others; stated confidence in the text is not a reliable signal of correctness.
Example
Prompt: "Give me three peer-reviewed papers on topic X with DOIs."
A model with no search access may return three references with realistic titles, authors and DOIs — some or all fabricated. The format is perfect, which is exactly why it is dangerous. The safe workflow: treat references as leads, then look every one up in a real index or publisher site before citing it.
When should I use it?
Hallucination risk should shape how you use AI, not whether you do:
- Low risk: brainstorming, drafting, rewording, code scaffolding you will run and test.
- Higher risk: facts, figures, citations, legal/medical/financial content, anything published or acted upon without review.
For higher-risk uses, add grounding (What is RAG (Retrieval-Augmented Generation)?), require sources, set up checks, and keep a human in the loop. Practical techniques are in How to Reduce AI Hallucinations; measurement is covered in How to Evaluate AI Systems and Outputs.
Common mistakes
- Believing a confident tone.
- Asking the model to check itself and treating that as verification. Self-review can help, but it can repeat the same error.
- Assuming RAG removes the problem. The model can still misread or ignore retrieved text, and retrieval can fetch the wrong passage.
- Assuming newer or bigger models are immune. Rates vary by model and task; test your own use case.
Try it with GoVibeTool
When you have a model's answer and the source text it should be based on, the Fact Anchor Checker compares each claim against the evidence you supply and flags what is supported, partially supported, or not found. It checks claims against your supplied evidence only — it cannot independently verify facts that aren't in that text.
Runs in your browser; the text you paste is not sent to a server.