Hallucination

The fact that models make mistakes is nothing new, and the output of any system can be wrong. What truly deserves separate discussion is that there is a category of errors that is entirely different in form from other errors: ordinary errors are often accompanied by hesitation—halting phrasing, vague wording, multiple contradictory alternatives, or simply admitting uncertainty. Hallucination, by contrast, is smooth, confident, and rich in detail; its fabricated content has an appearance almost indistinguishable from real content. A paper that does not exist at all will come with a complete title, authors, and journal name; an API that has never existed will be described with a name, arguments, and return values. Every detail reinforces the others, making it read like the truth.

The danger stems from this “too credible” appearance. Faced with an obviously hesitant or self-contradictory answer, readers are easily alerted and will actively verify it; but faced with a detailed, assured account, the impulse to verify drops sharply. It is hard to tell at a glance which sentence was generated based on facts and which was fabricated on the spot. The harm of hallucination is not that the model “got it wrong,” but that the error is packaged to be indistinguishable from correctness, so it can enter decisions, reports, or code undetected.

To determine whether a piece of output is a hallucination, it needs to be placed in a verifiable framework. This judgment process receives three inputs: the model output itself, the specific assertions within it that can be independently verified, and the authoritative evidence corresponding to them; the output is a status label for each assertion—“supported,” “contradicted,” “not covered,” or “unverifiable.” An assertion supported by evidence means it holds up; contradicted by evidence means it is a factual error; if the evidence scope contains no relevant content at all, it lacks a basis; if even the verifiable evidence scope cannot be determined, then it cannot be judged for the time being. Hallucination specifically refers to generated content that looks credible but lacks a basis or contradicts the facts; it emphasizes the “appearance of fabricated evidence.” An ordinary calculation error can also have serious consequences, but it usually does not carry this appearance of layer upon layer of fictional detail, so it does not belong to the hallucination pattern discussed here.

This judgment framework also implies a boundary: when the evidence scope is not specified, one cannot directly upgrade “not found for the time being” to “does not exist.” The fact that the model did not retrieve a certain fact only shows that there is no relevant content in the part of the evidence it saw; it cannot be used to conclude that the fact itself does not hold. “Not covered” and “contradicted” are two different states, and confusing them is itself a new source of hallucination.

A natural question: since hallucination is so harmful, why not just fix it as a bug? Why does it still exist as models get larger and larger? The answer is that it is not an implementation oversight, but stems from the nature of the model's training objective. What a large language model models is "the most likely continuation under the statistical patterns of the training data." The training process continually optimizes for "whether the next word looks like something a human would write," never "whether the statement is correct." The optimization direction is always language-level likelihood, never factual truth.

Two consequences follow directly from this. First, the model has no "truth database" internally to check against: when it generates each word, it relies on statistical associations between words in the training corpus, not on a query of the state of the world. Second, the model cannot perceive "I don't know": when it lacks a certain fact, it does not stop to report a knowledge gap, because the training objective has no item for "admitting uncertainty"; it just picks one of the statistically most plausible continuations and keeps writing.

Figure 1 illustrates exactly this. In both cases—"answering correctly" and "making things up"—the model does the same thing: it picks "the most likely next word." The only difference is that in the first case the training data happens to contain the correct answer, so the statistically most likely continuation just happens to be right; in the second case the training data has no answer, but the model still continues fluently under the same mechanism, generating content that is linguistically coherent but factually groundless. In other words, "answering correctly" and "making things up" have no dividing line in mechanism—they share the same generation process, just differing in whether the output happens to coincide with the facts. Precisely because the dividing line is not in the mechanism, we cannot expect to eliminate hallucination mechanically simply by scaling up the model.

A more precise formulation is: the pure language modeling objective does not provide a general-purpose "fact verifier." For open-world questions and long-tail facts, the model always has residual risk—as long as the correct answer to a certain type of question does not appear with sufficient strength in the training distribution, language likelihood cannot judge truth or falsehood on the model's behalf. Theoretical analysis can only provide lower bounds under specific distribution and calibration assumptions; there is no zero-error guarantee that holds for all questions. Scale, data, retrieval, tools, and post-training can indeed significantly reduce certain categories of hallucination—for example, having the model retrieve before answering on questions where evidence is available—but these measures reduce specific risk categories, not eliminate the structural root. Facing all possible unknown facts, no solution can provide a zero-error guarantee.

This structural risk analysis itself can be viewed as an input-output process: the inputs are the training objective, the current question, and the evidence available to the model; the output is a judgment about "whether there exists a factual gap that cannot be verified by language likelihood alone." The reason this gap necessarily exists structurally is that next-token training only increases the probability of plausible continuations based on context; it never consults a general-purpose truth arbiter. For long-tail or open-world questions, even with no evidence at all, it can still produce linguistically plausible continuations. Scale, data, and tools can narrow this gap, but cannot prove it is zero.

The previous section explained why hallucination cannot be eliminated at the mechanistic level, but what truly makes it difficult to guard against is a second feature: erroneous content is expressed with extreme certainty. Why does fabricated content come without hesitation and instead with a firm tone? Because “fluent, confident, and human-like” is precisely the surface feature that models are trained to approximate. The training signal rewards similarity to human expert text at the expression level, while “whether the content is correct” has never been part of that reward. The model learns to the extreme “how to phrase things like an expert,” but is never required to learn “how to show uncertainty when unsure.” The result is that it uses the same assertive tone for true content and for fabricated content—the tone itself carries no information about truth or falsehood.

A deeper reason is that when the model fabricates, it does not “know it is fabricating.” When a person tells a lie, they are inwardly aware of what the facts are and that they are deviating from them; the model has no such self-awareness. At the moment of generation, it is merely selecting the most probable words according to statistical regularities; from its internal process, this is no different from normal answering—it “thinks” it is answering normally. Since the model itself cannot distinguish the two, there is also no readable internal marker saying “this passage is my wild guess.” Therefore one cannot expect it to proactively label its own uncertainty, nor should one treat its self-statements—“I am very certain” and “as far as I know”—as evidence of credibility; these words are likewise text generated according to linguistic habits.

From this we can draw an actionable conclusion: judging the reliability of an answer cannot rely on how assertive its wording is. This can be formalized as a surface confidence analysis: input generation probability, expression style, evidence coverage, and calibration results, and output a judgment of “whether the strength of tone matches the evidence.” Training can make affirmative sentences more fluent, but it cannot turn assertive tone into actual probability—there is no correspondence between tone confidence and content accuracy that is guaranteed by the training objective. Dealing with hallucination depends on external verification and task-calibrated risk signals, such as whether evidence is sufficient and sources are checkable, rather than the confidence of the wording or the model's self-report.

Hallucination rates are not uniform across different types of questions. Although mechanistically it can appear at any time, several types of situations can significantly push the model from “answering with evidence” toward “completing according to language patterns.” Recognizing these signals is the most effective front-line defense in practical use.

The first type is content that is absent or very sparse in the training data. Obscure people, niche facts, your company’s private information—these are content the model has almost never seen during training. It has no statistical associations to rely on, yet still has to generate something, so it can only fabricate plausible-sounding details by following language patterns. Any question involving private data or long-tail facts is a high-risk zone for hallucination.

The second type is events that occur after the training cutoff. The model’s knowledge stays at the training data cutoff point; it knows nothing about subsequent news, policy changes, or product releases. But it will not refuse to answer just because it does not know; instead, it uses old knowledge to piece together text that appears to discuss the latest developments.

The third type is being induced by the wording of the question. When you ask “What is the conclusion of Zhang Wei’s paper?” the question itself presupposes the unverified fact that “Zhang Wei has a paper.” The model tends to fabricate along the lines of your presupposition—first assuming the paper exists, then filling in a title, conclusions, and details for it, rather than first questioning the presupposition itself.

The fourth type is high sampling temperature. Temperature controls the degree to which low-probability words are accepted during generation: the higher the temperature, the more the model “lets loose,” and the more likely it is to deviate from safe, high-probability answers and choose continuations that are linguistically flashy but factually dangerous. For the relationship between temperature and probability distributions, see the relevant section in “Information Theory and Entropy.”

Putting these conditions together, they can be organized into a high-risk condition judgment: the inputs are knowledge coverage, time range, presuppositions in the question, sampling temperature, and available materials; the output is the risk action that needs to be taken—retrieving supplementary evidence, clarifying presuppositions with the user, lowering the temperature, or directly refusing to answer. It should be noted that these conditions are only signals that increase risk, not an indication that an error will occur every time; conversely, low temperature cannot guarantee safety either—the same error may well be stably reproduced at low temperature. Risk signals are used to decide “when additional verification is necessary,” not to assert “this time it is definitely wrong.”

Hallucination cannot be eliminated, but its actual impact can be kept within a manageable range. All approaches revolve around the same idea: do not let the model generate out of thin air; instead, anchor it to verifiable materials and preserve an external verification channel for every critical assertion.

The first type of method is RAG (Retrieval-Augmented Generation): retrieve real materials before answering, and have the model answer based on the given sources. The model no longer generates solely from parameter memory, but is required to organize the answer from the text it has received, so the answer is "anchored" to real content. See the "RAG" deep-dive page for the specific mechanism. The second type is requiring citations: have the model indicate which material each sentence comes from, so that anyone who receives the answer can verify with one click. See "Citations and Source Attribution" for the specific practices of source tracing. The third type is decomposition, computation, and verification: for reasoning questions, require the model to list steps or call a calculator or code executor to compute. Note that the steps themselves can still be wrong, so critical conclusions cannot rely solely on the model's own reasoning; ultimately they still need to be verified with external evidence. The fourth type is risk screening and escalation to humans: after calibration and task validation are complete, you can combine answer consistency across multiple samples, retrieval coverage, scores given by the validator, or token probabilities to screen high-risk outputs, but raw logprobs are only generation probabilities and are not equivalent to factual correctness. The fifth type is prompting to allow "I don't know": explicitly tell it "If you are unsure, say you don't know; do not fabricate," which can reduce some fabrication because the model's compliance with the form of expression can partially compensate for its tendency not to proactively admit ignorance.

The common limitations of these methods must be made clear: they are all "mitigations," not "cures." If RAG retrieves the wrong thing, the material itself is wrong, or the question falls outside the scope of the provided materials, hallucination still occurs. As long as the underlying training objective is still "likelihood" rather than "truth," there is no once-and-for-all solution. The correct mindset is: verify every important fact, and never treat the model's confident tone as a guarantee.

The verification process for "Zhang Wei's paper" can fully demonstrate how this approach works. First, if you have the model generate directly, it only has the user's question assumptions and its own parameter memory; at this point you must never use that to fill in the paper title, journal, and DOI—any specific details are necessarily fabricated. Second, switch to searching academic indexes by author, year, and topic; the result has no exact match, but there are multiple authors with the same name. At this point the correct action is to downgrade the proposition "the paper exists" to "unverified," rather than picking the most similar-looking one from among the same-name authors to make do. Third, cross-checking the author's homepage and DOI registry still finds no match; you should explicitly report "not found" and at the same time list the search scope and the same-name ambiguity, so users can see the evidence boundary of the conclusion. Fourth, ask the user to provide the institution, title fragment, or link, turning "refusal to fabricate" into a next step that can continue gathering evidence.

Note the wording choices here: calibration does not mean softening your tone, but matching the strength of the assertion to the evidence coverage. "Not found in these sources" is more accurate than "this paper does not exist"—the former states the result of the search action, the latter jumps to a universal negative about the world; "may be a different author with the same name" is more verifiable than arbitrarily selecting a particular Zhang Wei. The inputs to the whole mitigation process are the question to be verified, retrieval sources, tool results, evidence coverage, and task risk; the outputs are an answer with citations, a scoped "uncertain," a clarifying question, or escalation to humans. RAG, citations, and prompting can all fail; any high-impact assertion must ultimately be personally verified with external evidence.

A longer answer is often a mixture of true and false: the first few sentences may be verifiable, while a key fabricated figure is inserted in the middle. If you only label the whole passage as “correct” or “wrong,” the most dangerous sentence will be averaged away. The correct approach is to first break the output into the smallest verifiable atomic assertions—each assertion is independently valid and independently verifiable—and then label each one according to the specified evidence set as one of four states: supported, contradicted, not covered, or unverifiable. What is being assessed here is the factual accuracy of assertions relative to the given evidence, not an exhaustive judgment of the truth about the whole world; for matters outside the evidence set, the judgment can only remain at “not covered.”

Use the “Zhang Wei paper” answer as a decomposition demonstration. One assertion is “the author's name is Zhang Wei”: there are too many authors with the same name in the evidence set, so it cannot be uniquely matched, and it is judged as not covered; the handling action is to request disambiguating information such as institutional affiliation. The second is “published in 2019”: likewise there is no evidence support, so it is judged as not covered; the handling action is not to fill in this year. The third is “titled ‘Graph…’”: the given DOI points to a different paper, directly constituting a contradiction; this is a critical error, and the entire answer must not be published. The fourth is “the topic involves graph neural networks”: this information comes only from the user's question presupposition, and the model cannot treat the user's words as a source; it should be marked as pending verification. The fifth is “I did not find an exact match”: this sentence is directly supported by the retrieval log and can be output, but the retrieval scope must be stated at the same time. As can be seen, sentences that appear in the same answer, after decomposition, have completely different handling methods.

With item-by-item determinations, you can calculate the atomic fact support rate: Rsupport = Nsupported ÷ Nverifiable, that is, the number of evidence-supported assertions divided by the total number of verifiable assertions. Suppose an answer contains 5 verifiable assertions, of which 3 are supported; the support rate is then 3 ÷ 5 = 60%. But the average score is not the only threshold: if the one contradictory assertion happens to be a critical field such as drug dosage, amount, or DOI, then even if the other four are all correct, the entire passage should be judged as a failure—the consequence of a dosage error is far from offset by an 80% support rate. Evaluation in production environments should simultaneously report support rate, critical contradiction rate, evidence coverage, and task coverage after model refusal, using multiple metrics to jointly characterize quality, rather than focusing only on a single average score.

The inputs to the entire atomic fact evaluation process are an answer and a specified evidence set; the outputs are the number of verifiable assertions Nverifiable, the number supported Nsupported, the support rate Rsupport, as well as the critical contradiction rate and coverage. The order of operations is: first break the answer down into the smallest assertions each of which can be independently verified, then map each assertion to evidence sources and determine its status, and finally calculate the ratio using Nsupported ÷ Nverifiable. 3 out of 5 supported is 60%; but a critical contradiction in a dosage, amount, or DOI is enough to fail the entire passage, which cannot be seen from the average support rate alone.

When a cited answer goes wrong, where exactly does the problem lie? “Having a citation” is not the same as “the citation supports this sentence”—the error may occur at any stage of the pipeline, and it must be attributed layer by layer before you can respond appropriately.

Moving from downstream to upstream, the first evaluation layer is retrieval coverage, whose core question is “Was the needed evidence retrieved?”. Typical failures include: the evidence never made it into the index, the query terms were constructed incorrectly, or same-name disambiguation failed and retrieved information about the wrong person. The second evaluation layer is evidence quality, whose core question is “Is the source primary, credible, and still valid?”. Second-hand summaries, outdated pages, and multiple sources copying each other—all of these put every subsequent judgment on an unreliable foundation. The third evaluation layer is faithfulness, whose core question is “Does the answer state only what the material actually supports?”. Typical failures include expanding “related” into causation, or writing “not mentioned” in the material as a negation of the fact. The fourth evaluation layer is citation correctness, whose core question is “Does each citation really point to the corresponding assertion?”. A common failure form is that the link itself really exists, but it does not support the adjacent sentence—the citation becomes decoration. The fifth evaluation layer is selective answering, whose core question is “When evidence is insufficient, should it refuse to answer or escalate to a human?”. A typical failure is forcing an answer in pursuit of answer coverage.

Behind these five layers are three goals that must be kept separate. Closed-book factuality asks “Is the answer consistent with the facts of the world?”—without external materials, this can only be answered from the model's own knowledge; faithfulness to a given material asks “Is the answer supported by the provided evidence?”; citation correctness asks “Does the link really support the adjacent assertion?”. The three are related but cannot replace one another: an answer with perfect citation formatting can be completely wrong factually, and a factually correct answer may cite the wrong source.

Layered diagnosis formalizes this troubleshooting process: input retrieval candidates, source versions, answer assertions, citation mappings, and refusal behavior; output a failure attribution to one of the layers—retrieval coverage, evidence quality, faithfulness, citation correctness, or selective answering. The specific problem it solves is: when an answer has citations, but the citations do not support the assertions, how to locate the error. The troubleshooting order is fixed: first ask whether the evidence was retrieved, then whether the source is reliable, then whether the statement exceeds the source, whether the link corresponds to the adjacent sentence, and finally whether it truthfully refuses to answer when evidence is insufficient. Only by locating the specific layer can the fix be targeted—if the index is missing, supplement the index; if disambiguation failed, revise the query; if faithfulness is insufficient, rewrite the generation constraints—rather than vaguely “regenerate it again”.

By stringing together the conclusions from the previous links in causal order, we can obtain a complete reasoning chain that starts from the single root of "next token likelihood" and extends all the way to what must be done in everyday use.

The starting point of the chain is the definition of hallucination: content that is fluent and confident yet lacks evidence or contradicts facts, which is harder to identify than ordinary wrong answers. It is hard to identify because it happens at the mechanistic level: the model models statistical likelihood of language rather than factual truth, and "answering correctly" and "making things up" are the same action inside the model—both are just picking the most likely continuation. This explains why it cannot be eradicated. At the same time, the model exhibits confidence unrelated to the truth or falsity of content, because "sounding human-written" is precisely the training objective itself, and the model cannot perceive what it does not know, so it uses the same assured tone for incorrect and correct content.

The mechanistic root determines the risk distribution: when training data are sparse, questions exceed the training cutoff, prompts contain false presuppositions, or sampling temperature is high, the model is more inclined to slip into pure language-mode completion, so hallucination occurs more frequently. Recognizing these signals is the basis for deciding "when verification is required".

Countermeasures unfold along two directions. First, anchor the model to real material before generation—RAG retrieval, requiring sentence-by-sentence citations, calling external tools for computation, supplemented by calibrated risk signals and human review. Second, perform structured verification on generated results: break long answers into atomic assertions, judge support, contradiction, non-coverage, or unverifiability item by item against evidence, and locate failures along five levels: retrieval coverage, evidence quality, faithfulness, citation correctness, and selective answering. Neither type of measure is a panacea—RAG can retrieve incorrectly, citations can be decorative, calibrated signals do not equal facts—so they must be used in combination, and ultimately all must rest on external evidence.

The end of the chain is a testable standard: being able to explain the core proposition that "language likelihood does not equal factual truth" and being able to design, for specific scenarios, a combined control of retrieval, citation verification, refusal, and human review is what counts as truly grasping the core of hallucination governance. Without reaching this step, any single-point tool merely hides the risk somewhere else.

Understanding this page requires three prerequisite concepts. The first is the 'large language model' itself—what it does and how it is trained; the entire analysis on this page rests on an accurate understanding of the language modeling objective. The second is 'predicting the next token', i.e., the way the model selects the most likely continuation word one by one; the structural root of hallucination is a direct consequence of this mechanism. The third is 'sampling and temperature', which determines the acceptance of low-probability words during generation, and is the premise for explaining 'hallucinations are more likely at high temperature' and how lowering the temperature reduces risk.

The core concepts of this page can be compressed into four groups: likelihood, not truth—the model optimizes for the plausibility of the continuation rather than the truth or falsity of facts; structural—hallucination therefore stems from the mechanism and cannot be eliminated by scaling up; confident tone—fluency and assertiveness are themselves training objectives and carry no truth/falsity information; mitigation, not cure—all measures such as Retrieval-Augmented Generation (RAG), citations, calibration, etc. only reduce risk and ultimately still rely on external verification.

There are five immediately adjacent extension directions. RAG is the process of retrieving first and then answering based on the given material before responding, and is the primary means of anchoring the answer in real content; citations and source attribution solve the problem of 'which sentence comes from where, and can it be verified in one click'; chain of thought lets the model explicitly write out the reasoning steps, making it easier to check step by step rather than only seeing the conclusion; token log probabilities and confidence discuss how to convert generative probability signals into calibrated risk estimates; retrieval underpins the evidence source for all the above schemes. One layer further out, the directions include: alignment—how to make the model's behavioral goals consistent with human intent; evaluation—how to systematically measure various failures including hallucination; interpretability—whether the model's internal representations can tell us why it answers this way; and reasoning models—a new generation of designs that trade longer deliberation for higher reliability. These directions are not substitutes for this page, but rather, built on the foundation of 'likelihood, not truth', each addresses a different aspect of hallucination governance.