Key answer
“Very certain” is part of the answer; “95% confidence” has an accuracy interpretation only after calibration on the relevant task. Numerical precision does not establish statistical measurement.
Full explanation
Calibration concerns groups of judgments: among many answers reported as 80% reliable, are approximately 80% correct? This is neither proof for one answer nor the probability of generating a word.
Self-reported confidence is not always worthless. Tian et al. compared GPT-3.5, GPT-4, Claude and others on TriviaQA, SciQ and TruthfulQA. With specific prompts, verbal confidence was better calibrated than estimates based on output probabilities in several settings, varying by model, dataset and elicitation method. Sections 2–3 and Tables 1–5 Validated confidence may prioritize reviews; casual chat percentages do not inherit that validation.
Ask what supports the certain portion, where uncertainty enters and what evidence would change the conclusion. “90% chance of success” could mean factual judgment, forecast or subjective confidence. Define the event, horizon and success criterion before evaluating a prediction.
Separate verified facts, assumption-dependent inferences and unresolved points. Give key evidence and uncertainty sources. If providing probabilities, identify statistical data, validated model estimates or subjective estimates.
Every sentence need not carry a percentage. Recalculate results, inspect original research and compare predictions with outcomes. For a classifier/QA system, test confidence on independent labeled samples, separately checking high-confidence accuracy and whether low-confidence flags capture important errors.