Hallucination

The survey by Ji et al. (2023) distinguishes intrinsic hallucination, where output contradicts the source it was given (a summary that reverses a figure in the document), from extrinsic hallucination, where output adds claims that the source can neither confirm nor refute. For open-ended assistants a further split is useful: factuality errors (the claim is false about the world) versus faithfulness errors (the claim is not supported by the provided context, whether or not it happens to be true). NIST AI 600-1 prefers the term confabulation and lists it as one of the risks specific to generative AI; OWASP's 2025 LLM Top 10 covers the downstream effect as LLM09 Misinformation.

The root cause is the training objective. Pretraining rewards assigning high probability to plausible continuations, and a fabricated citation with the right author-title-year shape is highly plausible text. Rare facts seen only once or twice in training are especially fragile, and the model has no separate store it can consult to check a claim. Post-training can make things better or worse: preference tuning that rewards confident, complete-looking answers, and benchmarks that score an abstention the same as a wrong answer, both give the model an incentive to guess rather than say it does not know - an argument OpenAI researchers made explicitly in 2025. Decoding matters too: higher temperature increases the chance of sampling a low-probability, wrong continuation, although greedy decoding does not prevent hallucination.

Hallucinated content typically looks locally coherent, which is why it is hard to catch. Well-known failure patterns include invented references and case law (the 2023 Mata v. Avianca case in New York, where lawyers were sanctioned for filing a brief citing non-existent decisions produced by a chatbot), fabricated software package names that attackers then register (slopsquatting), wrong numbers inside otherwise correct summaries, and confident answers about events after the knowledge cutoff.

Mitigations reduce frequency and make errors visible; none removes the phenomenon. Retrieval-augmented generation grounds answers in retrieved passages but introduces its own faithfulness failures when the model mis-reads or over-generalises a passage. Citation requirements with automated checks that each quoted span exists in the source, self-consistency checks across multiple samples, allowing and rewarding "I don't know", structured output for extraction tasks and tool use for arithmetic or lookup all help. Evaluation uses reference-based metrics, human review, or LLM-as-a-judge graders with their own error rates, so the residual risk has to be handled by process: a named human checks outputs before they are used in decisions, advice or code.