Uncertainty and Calibration: Making “80% Confidence” Really Mean About 80% Correct

Two assistants with the same overall accuracy of 80% can present completely different risks. Assistant A reports a confidence score of 0.8 for every question no matter what. Assistant B reports 0.95 on familiar cases and 0.35 on policy-conflict cases. If both ultimately answer 80% of the questions correctly, their accuracy is the same, but their usability differs greatly: if B’s reported scores are trustworthy, operators can route all low-confidence answers to humans, preserving most automation while significantly reducing the losses caused by errors. All of A’s answers carry the same 0.8, so they cannot be ranked by risk at all, and there is no way to triage. This difference does not show up in accuracy; it shows up in “whether the model knows how confident it actually is.”

Calibration is precisely about this question: whether the probabilities a model reports match the frequencies that actually occur. If over the long run the model answers about 80% of the questions it claims 80% confidence on correctly, it is well calibrated; if it reports 0.8 but only answers half correctly, it is overconfident. Here three abilities must be separated. Accuracy answers “how many were answered correctly”; discrimination answers “can it rank samples it will answer correctly ahead of samples it will answer incorrectly”; calibration answers a different question—“can the numerical values of the scores themselves be interpreted as real frequencies”. A model may be very good at ranking and generally place correct answers ahead of others, yet simultaneously give inflated probabilities for all samples; another model may be well calibrated but have low accuracy—for example, a model that always reports the base rate, whose probability commitments are naturally consistent with actual frequencies over the long run, but which provides no extra information for any specific case and has no triage value. Good calibration does not mean usefulness, good ranking does not mean the scores are interpretable, and the three are not the same thing.

From the perspective of evaluation inputs and outputs, calibration evaluation receives a set of task answers, the scores the model gives for each answer, and the final correct-answer labels; its output is a judgment: within groups with the same score, whether the empirical accuracy matches the probability the model claimed. It does not answer “how many did the model answer correctly”, nor “can the model rank”, but only “does reporting 0.8 correspond to being correct about 80% of the time in the long run”. In scenarios where low-confidence cases need to be routed to humans and resources allocated according to risk, the answer to this question determines the value of Assistant B—and also determines that Assistant A, which looks equally good, may not be safe.

Large language models (LLMs) do not naturally output “the probability that this answer is correct.” Phrasing generated by the model such as “I am very certain,” and the log probability (logprob) of a token, are not task accuracy. The former is just a linguistic style—the model imitates a confident tone without having been trained on probability; the latter is only the likelihood of the next word appearing given the preceding context, influenced heavily by length and wording, and has no necessary connection to whether the fact is correct. To turn model uncertainty into a usable score, you need to first construct a raw confidence from multiple signals, and then learn the mapping from “score → empirical accuracy” on independent data.

There are several types of raw signals that can be used, each with pros and cons. The log probability of a sequence or option is the cheapest and continuous, but is affected by length and phrasing and cannot be directly taken as factual correctness. The model's self-report is easily obtained through prompts, but it is essentially imitating a confident tone and lacks probabilistic meaning. Sampling the same question multiple times and comparing the semantic consistency of the answers can expose cases where the answer is unstable, but the same blind spot may cause all samples to be consistently wrong; high consistency does not mean correctness. External validation connects the answer to evidence or execution results and can provide an independent signal, but the validator itself also has coverage and error rates. Model ensembles use the disagreement of different models to reflect epistemic uncertainty, at the cost of being expensive, and different models may share the same origin with limited disagreement.

These signals can be combined. Taking a refund assistant as an example, you can combine “whether policy evidence was retrieved, the margin of the eligibility classification (the distance from the classifier's score to the decision boundary), the result of the amount program check, and the degree of disagreement among multiple sampled answers to the same question” into a risk score. But the score obtained at this step does not yet have probabilistic meaning: it can only say “the lower the score, the more suspect it is,” not “93% correct.” The same is true for average token probability; directly claiming “the answer is 93% correct” has no basis.

The step that truly endows probabilistic meaning occurs in the calibration phase: on a calibration set independent of the training data, align raw scores with actual correct frequencies, and learn the mapping from scores to empirical accuracy. Here, the task event for “correct” must first be clearly defined, for example “the refund eligibility judgment is correct and cites evidence supporting that judgment.” The inputs are raw signals such as token probabilities, multiple samples, model self-evaluation, retrieval coverage, external validators, and model disagreement; the output is a calibrated probability that can be interpreted as correct frequency only after the mapping. Before this, any “I am very certain” or average token probability given by a single path can only serve as a risk clue and cannot be directly used as the probability that the answer is correct.

Once we have a probability score for each prediction, how do we check whether a batch of scores between 0 and 1 lives up to their names? The core approach is to bucket predictions by confidence, then compare two things within each bucket: the bucket's average confidence, conf(Bₘ), and the actual accuracy of the samples in that bucket, acc(Bₘ). If the cases where the model reports 0.8 actually answer 80% correctly, and those where it reports 0.9 answer 90% correctly, then points drawn with “average confidence” on the horizontal axis and “actual accuracy” on the vertical axis should all fall on the y = x diagonal—this is the ideal reliability diagram. If the curve lies overall below the diagonal, the model is systematically overconfident: the probabilities it promises are higher than the frequencies it delivers; if above the diagonal, the model is overconservative, and its actual performance is even better than it claims.

Bucket-wise comparison has an inherent flaw: structure inside a bucket is hidden. A single 0.7-to-0.8 bucket may mix samples with true probabilities of 0.6 and 0.9; as long as the average happens to fall on the diagonal, the plot shows no problem. Therefore reliability diagrams need to be read together with continuous calibration methods and key slices.

Expected Calibration Error (ECE) summarizes the deviation on a reliability diagram into a single number. For each bucket, compute the absolute difference between average confidence and actual accuracy, then weight it by the bucket's sample proportion and sum:

ECE = Σₘ (|Bₘ| / n) × |acc(Bₘ) − conf(Bₘ)|

Here, n is the total number of predictions, Bₘ is the m-th bucket, and |Bₘ|/n is that bucket's sample proportion. The smaller the ECE, the smaller the average gap between probability promises and actual frequencies under the current bucketing scheme. But the ECE value depends directly on the number of buckets and where their boundaries are placed: for the same batch of predictions, cutting it into 10 buckets versus 20 buckets will produce different ECE values, so ECE should not be mechanically compared across different bucketing settings. In addition to ECE, there are metrics with different penalty structures: the Brier score (p − y)² penalizes both probability deviation and prediction error; negative log-likelihood penalizes “high-confidence but wrong” more heavily, because −log(p) in that situation becomes very large.

More importantly, calibration is conditional. An overall nice diagonal plot does not guarantee that all scenarios are well calibrated: subsets split out by Chinese-language orders, extremely long orders, or newly launched policies may still each be severely overconfident, only hidden by the overall average. Reliability evaluation must be checked in groups conditioned on factors that affect risk. To sum up, the inputs to the evaluation are n predictions and their buckets Bₘ, and the outputs are a reliability diagram and an ECE value; for each bucket, take the absolute difference between confidence and accuracy, multiply by the sample proportion, and sum to obtain the average calibration error under the current bucketing. Interpreting the result requires attention to two points: the value depends on bucket boundaries, and bucket-level aggregation may hide miscalibration at the slice level.

Place the model scores and final correctness labels for ten refund requests into three buckets, and you can walk through the ECE calculation in full. The reliability diagram for the three confidence buckets shows that the high-confidence bucket actually got only two-thirds correct, exposing overconfidence; another message from the diagram is that the human handoff threshold must be set based on calibrated risk, not raw scores.

Bucket statistics are as follows: the low-confidence bucket has 3 samples, average confidence 0.30, 1 correct, accuracy 0.333; the medium-confidence bucket has 4 samples, average confidence 0.65, 3 correct, accuracy 0.75; the high-confidence bucket has 3 samples, average confidence 0.90, 2 correct, accuracy 0.667. Each bucket's weighted difference is “the absolute difference between accuracy and average confidence” multiplied by that bucket's sample proportion. Low bucket: 3/10 × 0.033 ≈ 0.010; medium bucket: 4/10 × 0.10 = 0.040; high bucket: 3/10 × 0.233 ≈ 0.070. Adding the three contributions gives ECE ≈ 0.120.

How should this number be read? First, with only ten samples, the estimate is extremely unstable; 0.120 is only a demonstration calculation, not a precise production estimate. Second, the operationally important finding lies in the highest-confidence bucket: the model reports an average confidence of 0.90 on these cases, but only two-thirds are actually correct. If the cost of an incorrect refund is high, then treating “auto-approve above 0.7” as a safe threshold by no means holds—the raw score may look high, but after bucketing this is exactly what is exposed. The calculation rule itself is simple: for each bucket, take |accuracy − average confidence| multiplied by the sample proportion, then add up the contributions of all buckets; the input is the scores and correctness labels of ten refund requests, and the output is the individual contributions of the low, medium, and high buckets plus the total ECE. The high bucket has average confidence 0.90 but only 2/3 correct, showing that the original threshold would let through many high-cost errors—this is exactly the message that calibration evaluation should convey to operational decisions.

Classification probabilities from models are often systematically over-sharp: nearly all predictions sit at 0.95 or above, even when the actual error rate is far higher. Without retraining the model, how can such scores be pulled back to reality? Temperature scaling is one of the lightest-weight approaches. Let the logits output by the classifier be z; temperature scaling recomputes the probabilities as softmax(z/T):

pᵢ = exp(zᵢ / T) / Σⱼ exp(zⱼ / T)

Here T is the temperature parameter, chosen on an independent calibration set to minimize the calibration set's negative log-likelihood. When T is greater than 1, the exponents are compressed, the distribution becomes softer, and overconfidence decreases; when T is less than 1, the distribution becomes sharper and the model appears more confident. The key property is that scaling does not change the ordering of the class scores, so it usually does not change the top-1 prediction or accuracy; it only adjusts the probabilistic meaning of these scores — it preserves 'which class has the highest score' and corrects 'what real-world frequency this confidence corresponds to.'

Temperature scaling is a global linear correction; it can only soften or sharpen the distribution as a whole. In binary classification settings, the simpler Platt scaling can be used; if the bias is severely nonlinear, isotonic regression can fit an arbitrary monotonic mapping, but it is more prone to overfitting when data are scarce. For free-text generation, the issue is even more upstream: you must first clearly define the event to be calibrated—for example, 'the eligibility judgment is correct and the cited evidence supports that judgment'—and then calibrate the score for that event. Without a clear definition of the correct label, it is unclear what a reported 0.8 refers to.

Regardless of which correction is used, data discipline is indispensable: the training set learns the model, the validation set tunes the system, the calibration set learns the 'score → probability' mapping, and the test set evaluates final performance. Using test-set labels to select the temperature is equivalent to leaking the test set into parameter selection, and the calibration results will again carry optimistic bias. The inputs to temperature scaling are the classification logits z, the independent calibration labels, and the temperature T; the output is the rescaled probabilities softmax(z/T). It only fixes the score scale and does not create knowledge that the model did not originally have. If the event definition changes, or the online data distribution changes, you must re-verify that the calibration still holds.

Calibration gives a risk probability for each answer; next you need to decide what to do for each request: answer automatically, route to a human, or reject. Selective prediction works by setting a threshold τ and automatically answering only samples whose score is at least τ. This produces two competing metrics: coverage, the proportion handled automatically, and selective risk, the error rate among automatically handled samples. As τ increases, coverage typically falls, while the risk in the automatically admitted portion also falls. Evaluation should plot the full risk–coverage curve rather than reporting only numbers at some convenient threshold—the curve reveals in which threshold region each unit of coverage given up buys how much risk reduction.

Where exactly to set the threshold depends on the cost structure. We can write the expected cost of a threshold policy as:

Cost(τ) = P(auto | τ) × P(error | auto, τ) × Cerror + P(manual | τ) × Cmanual + P(reject | τ) × Creject

That is, multiply the probability of each of the three outcome types—automatic, manual, reject—by its corresponding cost and sum them: the cost of automatic handling comes from the error cost Cerror of the erroneous cases among them, manual handling incurs manual cost Cmanual, and rejection incurs rejection cost Creject. Taking refunds as an example, if the cost of wrongly promising a refund is 100 and a single manual review costs only 4, then even if many cases go to manual handling the total cost may still be worthwhile; by contrast, for low-risk copy polishing, error cost is very low, so the threshold can be relaxed and more requests can be handled automatically. Besides cost there are real-world constraints: manual review capacity and SLA. When the queue is full you cannot quietly auto-approve high-risk cases; it is better to degrade to delayed handling or explicit rejection than to let the system compromise on the high-risk threshold.

Multi-stage systems can also split a single threshold into multiple gates, because a single answer confidence cannot cover all risk sources: when evidence is insufficient, supplement retrieval before deciding; when amount validation fails, route to a human; when permissions conflict, prohibit the action. Each gate manages one kind of risk, which is more operational than a single overall confidence score used as a blunt cutoff. The inputs to the entire selective decision are the calibrated score, threshold τ, error cost Cerror, manual cost Cmanual, rejection cost Creject, review capacity, and SLA; the outputs are the disposition for each request (automatic, manual, or reject) and the expected cost Cost(τ) under the threshold. Choosing the operating point means, subject to capacity and SLA constraints, picking a τ that makes the expected cost acceptable—the risk–coverage curve and the cost formula respectively answer 'how much automation to give up' and 'how much automation is worth giving up'.

The model can report a high conditional probability for every token it generates, yet the whole answer can still be a hallucination. The reason is that sequence probability is the product of the conditional probabilities of the individual tokens, and longer sequences naturally have smaller products, so answers of different lengths cannot be directly compared; switching to average log probability is also problematic: it systematically favors common, short, templated phrasings over factually correct phrasings. The truth of a refund conclusion may hinge entirely on one entity name or one monetary amount, and that token is linguistically extremely common—highest probability, most factually wrong. Multiple sampling is equally unreliable: several samples giving the same wrong answer may merely be a shared blind spot induced by the training data and prompt style, not evidence that “the model is very confident.”

Therefore the object of calibration must shift from “text fluency” to “task events.” Whether the eligibility conclusion is correct, whether the citation entails the conclusion, whether tool execution succeeds, and whether the final business state matches expectations should each be defined with separate labels and calibrated separately; treating the probability of the entire generation as the overall correctness rate is measuring the wrong thing.

Several typical failure modes deserve separate handling. Length bias manifests as inflated scores for shorter answers and is corrected by length normalization and task-defined events. Same-source blind spots manifest as consistent errors across multiple samples and require external evidence and heterogeneous verifiers to break the consistency illusion. Slice mismatch manifests as good overall calibration but severe miscalibration for minority slices and must be checked using stratified reliability and minimum sample requirements. Distribution drift manifests as a sudden surge in errors under old thresholds and can only be addressed through continuous monitoring, recalibration, or proactive degradation.

The inputs to sequence confidence analysis are token conditional probabilities, text length, semantic sampling results, task event labels, and external evidence; the output is a risk score defined in terms of task events. Its boundary is also clear: out-of-distribution inputs, new policy rollouts, retrieval index switches, or adversarial prompts all invalidate the old “score → correctness rate” mapping. In these cases, high confidence reported by the model cannot constitute a safety proof; it is only a residual reading from an old mapping that has not yet been retested.

Calibration is not a one-time task completed at launch: ground-truth labels often arrive days or even longer later, while online risk is real-time. The first thing production monitoring must do is make evaluation possible—save the raw score, calibration version, threshold, disposition outcome, task slice, and final label for every decision. With these records, once labels mature at any later point, you can go back and recalculate.

When labels are not yet mature, use proxy signals for early warning: missing evidence, tool execution failures, user corrections, and human overrides arrive faster than final labels and can indicate that a slice may be deteriorating. But they are only early warnings and cannot replace calibration metrics; after labels mature, recalculate ECE, Brier score, and risk–coverage curves. Metrics should be viewed with rolling windows and confidence intervals to avoid frequently changing thresholds due to normal variation from small samples.

The threshold itself should be set and frozen before release: select the threshold on a frozen evaluation set, record the expected coverage and risk as the post-launch comparison baseline. In the early post-launch period, sample in a canary fashion and manually review cases that were automatically passed, especially the batch whose scores sit right at the threshold boundary—that is where threshold error is most concentrated. Monitoring must be performed by slice: language, policy version, and input length should be examined separately; do not look only at aggregate numbers, otherwise a few slices being off will be averaged away.

When drift or high-confidence errors exceed budget, actions should be explicit: tighten thresholds, roll back the old calibration mapping, or pause automatic actions. The last point is easy to get wrong: recalibration requires new, representative labels. Simply increasing the temperature without new labels to make the dashboard ECE look good does not restore consistency between event definitions and the data distribution; it merely changes the scale on an already-invalid mapping. The inputs to production monitoring are raw scores, calibration version, thresholds, decisions, slices, and delayed-maturing labels; outputs are rolling ECE, Brier, risk–coverage curves, and the disposition actions of tightening, rolling back, or pausing. Proxy signals only serve an early-warning role.

Connect the preceding steps in causal order; calibration from a problem to verifiable practice has six steps.

The first step is to define the correctness event to be calibrated. Without this step, none of the subsequent probabilities would know what they refer to: what needs to be calibrated is “the refund eligibility determination is correct and supported by citations,” not “the text is fluent” or “the average token probability is high.” The event definition also determines how labels should be assigned.

Step Two obtains raw scores from the model, evidence, and verifiers. Token log probabilities, consistency across multiple samples, model self-assessment, retrieval coverage, external validation results, and model disagreement can all be combined as signals, but at this stage the scores are only risk clues and do not yet have probabilistic meaning.

Step Three learns the mapping “score → empirical accuracy” on an independent calibration set. This mapping is valid only for the signals from Step Two and the event definition from Step One, so the calibration set must be independent of the training and test sets.

Step Four uses reliability diagrams and scoring rules for acceptance. After binning, compare the average confidence with the actual accuracy, compute ECE, and when necessary use Brier score or negative log-likelihood to add a penalty perspective, and check whether key slices are miscalibrated. This step answers whether reporting 0.8 really corresponds to about 80% being correct.

Step Five chooses thresholds by error cost. With interpretable probabilities, use expected cost to trade off coverage against risk: if error cost is high, tighten the threshold and route more to human review; if error cost is low, relax the threshold and automate more, while remaining constrained by review capacity and SLA.

Step Six continuously collects mature labels and monitors drift. After deployment, save the score, threshold, slice, and final label for each decision; once labels mature, recompute calibration metrics on a rolling basis; if new policies, new language slices, or distribution drift cause high-confidence errors to exceed the budget, tighten thresholds, roll back mappings, or pause automated actions, then wait for new representative labels to recalibrate.

The six steps are interlocking: the event definition determines the semantics of scores and labels, the independent calibration set turns scores into probabilities, reliability acceptance verifies probability commitments, cost thresholds turn probabilities into disposition decisions, and production monitoring ensures this chain still holds after distribution shifts. If any step breaks—ambiguous event definition, calibration set leakage, reporting only overall results without checking slices, setting thresholds on raw scores, using temperature scaling instead of recalibration—then “80% confidence” degenerates back into a phrase with no frequency meaning.