Self-consistency: Voting with Multiple Independent Reasoning Paths

Large models generate text token by token. When solving multi-step reasoning problems, the model outputs step by step along a chain of thought, and each step is conditioned on all the content generated so far. Therefore, an early reasoning choice—which formula to use, which branch to take, which quantity to treat as known—determines the direction of all subsequent steps: if the early choice is wrong, everything that follows continues along the wrong path, and the final answer goes wrong as a result.

The key is that sampling-based decoding itself involves randomness. With the same model and the same problem, sampling again may lead to different branch choices early on. This is exactly the answer to “why the same model may get it right with a different sample”: the correct path is not beyond the model's reach; the previous sample simply did not happen to select it randomly.

Self-consistency turns this observation into a method: sample the same multi-path-solvable problem independently multiple times, producing multiple candidate reasoning paths. If the correct path appears more often in the sample set than any single wrong path, then taking a majority vote on the final answers can reduce the impact of a single path's occasional error—a single early random branch mistake only contaminates the path it is on, while the more frequent correct path has a chance to win the vote. Self-consistency targets the occasional error of a single path.

The inputs to self-consistency include: a problem that can be solved through multiple paths, sampling configuration (temperature, number of samples, etc.), and aggregation rules (such as majority vote on normalized answers). The outputs include: multiple reasoning paths, normalized answer clusters, and the aggregated result.

Its boundaries are equally clear: if the majority of sampled paths share the same bias—for example, they all believe the same wrong premise—then the vote will still go wrong. Self-consistency can only correct random errors scattered across individual paths; it cannot fix a wrong premise that all paths jointly believe.

Self-consistency treats “whichever answer the majority of paths agree on” as the final answer; its reliability can be characterized by a simple voting model: each path has an independent probability p of being correct, we sample n paths in total (n is odd to avoid ties), and we ask for the probability that the majority vote is correct.

If the errors of the paths are approximately independent, then the number of correct paths among the n paths follows a binomial distribution with parameters (n, p). Majority correctness means that the number of correct paths is at least (n+1)/2, so

P(majority correct) = Σ from k=(n+1)/2 to n of C(n,k) × p^k × (1−p)^(n−k)

This expression explains all the key phenomena. When p > 0.5, the distribution of the number of correct paths is concentrated around np > n/2; as n increases, the probability that the majority falls on the correct side increases monotonically and approaches 1. Voting amplifies the small advantage of a single path into near-certain correctness. Conversely, when p < 0.5, the distribution is concentrated below n/2, and the larger n is, the more certain the majority is to be wrong—voting amplifies the signal itself; if the signal is wrong, it amplifies error. If the errors of the paths are highly correlated, for example if they share the same set of wrong premises, then each additional sample carries little new information, the gains from increasing n quickly saturate, and voting does not reliably improve.

Therefore, the inputs of this probability intuition are the single-path accuracy p, the odd sample size n, and the path independence assumption; the output is the probability that the majority vote is correct. The conclusion splits into three cases: when p > 0.5 and errors are approximately independent, increasing n usually improves the majority accuracy; when p < 0.5, increasing n only strengthens the error; when error correlation is high, the benefit saturates with n.

This conclusion describes the probabilistic behavior under an idealized voting model. It relies on the assumption that paths are approximately independent, and p itself is not directly observable; it indicates the conditions under which voting is worth doing, rather than guaranteeing the factual correctness of any specific answer.

Self-consistency requires multiple mutually different reasoning paths, and the source of path differences is the randomness in sampling decoding. Temperature is the most direct knob for controlling this randomness: it acts on the probability distribution of the next word output by the model. The lower the temperature, the sharper the distribution, and the model almost always selects the word with the highest probability; the higher the temperature, the flatter the distribution, and low-probability words also have a chance to be selected.

Temperature of zero means deterministic decoding. At each step the word with the highest probability is chosen, the same path is repeated verbatim, sampling ten times yields ten identical inferences, voting degenerates into a single path, and self-consistency loses its meaning. After moderate heating, early branches begin to diverge; correct paths and various incorrect paths both have a chance to be sampled, giving voting room. But when temperature is too high, the distribution approaches uniform, the model selects many words that are unreasonable in the reasoning context, producing broken steps, invalid derivations, or unparseable answers, and the quality of the candidate set collapses. There is tension between diversity and quality: too cold means no diversity, too hot means no quality.

Another common knob is top-p sampling, which truncates the candidate words to the smallest set whose cumulative probability reaches p, and samples only within this set. Temperature reshapes the entire distribution's shape; top-p cuts off the long tail. The two act differently and usually need to be considered together with the number of samples. In engineering, one does not tune a single parameter in isolation; instead, on the task set one jointly tunes temperature, top-p, and the sample count upper bound, and continuously monitors two signals: the number of unique answers after normalization, and the proportion of parse failures and reasoning interruptions. If the former is too low, it means the paths have not truly separated; if the latter is too high, it means randomness has already harmed quality.

The inputs of this stage are temperature, top-p, the sample upper bound, the per-path validity rate, and answer diversity; the output is a set of candidate paths that are quality-usable and mechanistically different. The purpose of diversity is to make the errors of the paths as uncorrelated as possible, so that voting approaches the case of independent samples; it does not mean the more random the better. The randomness beyond what is needed for decorrelation only consumes more sampling budget and produces invalid inference.

The object of voting is the final answer, not the free-form text written down by each path. Different reasoning paths may express the same answer in different surface forms. Direct string-based vote counting would split the same answer into multiple candidates, so before counting, answers must be normalized into comparable equivalence clusters.

The normalization approach depends on the task type. For math problems, you can first parse the structure of the final answer, then simplify and convert it according to rules: 1/2 and 0.5 are two notations for the same numerical value; 50% and 0.5 are expressions of the same proportion; 0.5 m and 50 cm are two units for the same length; answers that differ on the surface but have the same meaning should be placed into the same cluster. For classification tasks, the label set is fixed, so normalization means mapping each path's answer to a unique label, and the mapping rules are themselves equivalence rules. Open-ended question answering has no finite label set, so it requires semantic clustering or a verifier to determine whether two expressions are equivalent—for example, using embedding similarity or having the model judge each pair.

The granularity of normalization directly determines how many votes each answer ultimately receives. If the granularity is too fine, equivalent answers will be split apart; clusters that should have been merged are dispersed into multiple clusters, correct votes are diluted, and an incorrect answer with fewer votes may win. If the granularity is too coarse, actually different answers will be merged into the same cluster, artificially inflating that cluster's vote count. Errors in the aggregator itself appear in the same two forms: incorrect splitting or incorrect merging.

The inputs to this step are each path's final structured answer, units, numerical form, and task equivalence rules; the outputs are equivalence clusters eligible for voting and a record of parsing failures. Answers that cannot be parsed into a structured form cannot enter vote counting and should be counted and monitored separately, because they reflect the lower bound of upstream sampling quality. Normalization rules are always task-specific; there is no universal conversion table. The same set of rules may be correct for one task and, for another, may create split votes or merged votes.

Ten paths that give the same answer only show that this sampling process has a stable preference for that answer, not that the answer matches real-world facts. Voting measures the internal consistency of model behavior, and the source of errors often lies in the layer shared by all paths: the same bias in the training data is inherited by every path; the same misremembered formula is repeatedly invoked in every derivation; a certain wording cue in the prompt also pushes all paths in the same direction. These correlations make “ten paths” informationally close to “one path repeated ten times”: the vote count is high but the amount of evidence has not increased.

To break this kind of correlated error, what needs to change is not the number of samples but the source of the paths. An external calculator can directly verify arithmetic steps, retrieval can check factual statements, unit tests can verify the constraints that code paths claim to satisfy, and independent rules or another model can cross-check reasoning premises. These external checks differ from “sampling one more path from the same model”: their errors do not share the same source as the model’s training bias, so they carry genuinely new information. For samples from the same model, increasing the count cannot remove shared bias; changing the source or introducing verification tools is the more effective upgrade.

Therefore, interpreting a consensus result requires considering both the vote count and the source of the paths. The input includes not only the vote distribution but also whether the paths come from the same model or multiple models, whether the prompt is uniform, and whether external evidence exists; the output is a judgment of the strength of the consensus and a decision on whether external verification or refusal is needed. A high-impact conclusion should not end with “most paths agree” but should be given only after verification by calculation, retrieval, tests, or independent rules.

Sampling n times means paying n times the inference cost, and the marginal benefit of each additional path decreases as n increases: once the probability of the majority being correct approaches its upper bound, further sampling brings only tiny improvements. Therefore, “fixed sampling of 40 times per problem” is usually not a good default setting—it makes simple problems pay dozens of extra inference cost for nothing, and may leave hard problems still without enough vote margin within 40 samples.

A more reasonable approach is sequential sampling: sample a small batch of paths at a time, count votes, then decide whether to continue or stop based on the current state. There are several stopping conditions: the external verifier has passed, indicating that the answer holds regardless of vote count; the vote margin is already large enough to make a decision under the given confidence requirements; or the expected benefit of continued sampling is lower than its cost, or the remaining budget is exhausted. Conversely, if the vote margin is small and verification has not passed, continue adding samples or give up answering. The inputs to this decision include the current vote margin, effective sample size, verification result, remaining budget, task value, and error cost; the output is one of five actions: continue sampling, stop and answer, verify the leading answer, refuse to answer, or escalate to a human.

Enabling self-consistency itself should also have a threshold: enable it only for tasks that are high-value, where answers can be normalized and aggregated, and where a single run is unstable (multiple samples of the same problem produce scattered answers); otherwise aggregation has no basis and the cost yields no return. After enabling it, you should report a quality–cost curve, i.e., how accuracy changes with the number of samples, so that the budget spent is well documented.

Also note that vote proportion is an uncalibrated signal. An 8/10 agreement only means that eight out of ten paths agree, and does not equal an 80% probability that the answer is correct; shared bias and path correlation prevent vote proportion from being directly mapped to confidence. Therefore, stopping decisions should rely on vote margin and verification results, not on treating vote proportion as a probability.

Assume each path is independent, the single-path accuracy is p = 0.6, and five paths are sampled. What is the probability that the majority (at least three) are correct? The input to this hand calculation is the independent single-path accuracy p = 0.6, and the output is the probability P_majority that at least three of the five are correct.

“At least three correct” consists of three mutually exclusive cases: exactly three correct, exactly four correct, and exactly five correct. The probability of each case is computed term by term with the binomial formula: exactly k correct has C(5,k) combinations, and each combination has probability 0.6^k × 0.4^(5−k).

Exactly three correct: C(5,3) = 10, 10 × 0.6³ × 0.4² = 10 × 0.216 × 0.16 = 0.3456 Exactly four correct: C(5,4) = 5, 5 × 0.6⁴ × 0.4 = 5 × 0.1296 × 0.4 = 0.2592 Exactly five correct: C(5,5) = 1, 1 × 0.6⁵ = 0.07776

P_majority = 0.3456 + 0.2592 + 0.07776 = 0.68256 ≈ 68.3%

The majority accuracy improves from 60% for a single path to about 68.3%. The improvement is real, but far smaller than fivefold. The marginal contribution of each vote decreases with n; voting gradually amplifies the weak accuracy advantage of a single path rather than multiplying reliability by the number of paths.

Replace p with 0.4 and calculate again: the same three terms add up to 0.2304 + 0.0768 + 0.01024 = 0.31744 ≈ 31.7%. When a single path is slightly biased toward error, the majority is actually more likely to stand on the wrong side—sampling is not inevitably beneficial; the direction depends on whether p is greater than 0.5. If the errors of the five paths are highly correlated, for example sharing the same incorrect premise, then they carry far less independent information than five independent samples, the effective sample size is less than 5, the formula above no longer applies, and the actual majority accuracy will be noticeably lower than 68.3%.

The conclusions for the three conditions can be summarized as follows: when p > 0.5 and errors are relatively independent, majority accuracy increases with sample size; when p < 0.5, the majority is more likely to be wrong; when errors are highly correlated, the benefit of adding samples saturates quickly.

Figure 1 depicts the complete Self-consistency flow as a left-to-right auditable chain, and explains why “sampling ten chains of thought” is not the same as “ten independent witnesses.”

The starting point of the chain is the same question, which branches downward into multiple sampling paths. Before aggregation, the first step is to extract the final answer of each path, rather than retaining the entire reasoning text: no matter how long the internal reasoning text is, it does not add weight to that vote; the object of voting is only the extracted answer. The second step is normalization and vote consolidation, merging answers that are superficially different but semantically identical, such as 1/2 and 0.5, into the same cluster, to ensure that votes for equivalent answers are not split. The third step is voting, producing the majority result. The process does not end at voting: the majority result still needs external verification—using computation to check numerical values, retrieval to check facts, tests to check code, and independent rules to check constraints—after which the final answer is output, together with the verification status and tracking records for each stage.

The input of this figure is multiple paths from the same question, an answer normalizer, a voter, and an external verifier; the output is the final answer, verification status, and stage tracking. What it expresses is exactly this causal chain: multiple paths only provide candidates and vote counts, normalization determines how votes are combined, and external verification determines whether the majority result is credible; all three links are indispensable. The reason it is not ten independent witnesses is that all paths come from the same model, share the same set of training biases and sampling process, and are highly correlated statistically; the step in the figure from voting to external verification is precisely the response to this correlation. The entire figure describes an auditable search process, not ten unrelated bystanders.

When all five paths invoke the same erroneous formula, voting does not help at all, because their errors are correlated: shared model weights, the same training data, the same prompt, and similar decoding prefixes make these paths tend to make errors together and make the same kind of error. Voting relies on the assumption that errors are approximately independent; correlated errors directly weaken that assumption.

The effect of correlation on the benefit of voting can be quantified by variance. Let the variance of the answer error for a single path be σ², and let the correlation coefficient between the errors of any two paths be ρ. Then the error variance after equally weighted averaging of n paths is approximately σ² × (1 + (n−1)ρ) / n. When ρ = 0, the variance equals σ²/n, decreasing as 1/n—this is exactly the benefit of independent voting; when ρ = 1, the variance equals σ², and no matter how large n becomes it does not decrease at all—all paths give the same bias, and averaging cannot eliminate it. A ρ between the two indicates a discounted benefit.

Converting the discounted variance into “how many independent samples it is equivalent to” yields the approximate formula for effective sample size:

n_eff = n / (1 + (n−1)ρ)

Taking n = 10 and ρ = 0.5 as an example, n_eff = 10 / (1 + 9 × 0.5) = 10 / 5.5 ≈ 1.82. Nominally ten paths were sampled, but in practice they only achieve the variance reduction of about 1.82 independent samples; most of the extra inference cost is swallowed up by the correlation.

The inputs of this analysis are the nominal sample size n, the path error correlation coefficient ρ, and the single-path variance, and the output is the effective sample size n_eff. Ways to reduce ρ are to increase mechanism diversity: use different prompt decomposition approaches, introduce different tools, use different model families, and add retrieval evidence so that the error sources of the paths no longer overlap. At the same time, prevent the aggregator itself from creating bias; for example, if a voting rule favors verbose answers or answers from the same source, it effectively introduces new correlation artificially. Also remember that ρ is only an approximate summary: real correlations between paths will not be exactly described by a single number, and n_eff is a model for understanding benefit saturation, not an exact sample count.

Fixed sampling counts treat easy and hard problems alike, whereas sequential sampling turns “when to stop” into an online decision: first take a small batch of samples, normalize answers and count votes, then check the stopping signal; if it is triggered, stop; if not, sample another batch, until the budget limit is reached.

There are four types of stopping signals, each with different benefits and risks. Fixed vote margin is the simplest: stop when the vote margin between the leading answer and the second-place answer reaches a preset threshold, but it completely ignores correlations among path errors—no matter how large the margin is, it may still be a repetition of the same bias. Sequential statistical bounds provide explicit error control, using statistical bounds to judge whether there is enough confidence to make a decision, but their guarantees rely on assumptions such as independence; when the assumptions fail, the bounds break down. Verifier pass is one of the strongest signals because it introduces external evidence beyond vote counts, using computation, retrieval, testing, or rules to verify the leading answer, but the verifier itself may have blind spots. Marginal benefit signals directly trade off cost and benefit, stopping when the expected benefit of the next batch of samples falls below its cost; the cost is that the benefit must be estimated online, and the estimation itself introduces noise.

The inputs to sequential sampling are the current normalized vote counts, correlation estimates, external verification results, frozen stopping boundaries, and budget; the output is one of five actions: continue sampling, stop and answer, verify the leading answer, refuse to answer, or escalate to a human. Among these, the thresholds must first be tuned on a frozen development set according to task value and error cost, and must not be changed after deployment; any usage that treats vote share directly as a probability will introduce uncalibrated confidence. When assumptions such as independence do not hold, signals of the fixed-vote-margin type do not provide any error guarantee; in that case, it is better to rely on external verification or escalation to a human rather than continuing to sample more.

A drug dosage conversion problem can fully demonstrate how self-consistency makes step-by-step decisions among quality, risk, and cost. The input to the entire process is a problem that requires a “numerical value + unit” output, the necessary variables required by the problem, the first batch of three sampling paths, unit rules and range-check rules, and a budget of up to five paths; the output is a structured answer, validation result, or expert escalation.

First step, first define the answer space. The final output must include a numerical value and a unit, so before any sampling, establish the unit conversion relationships between mg and g, and between daily dose and per-dose. If the problem lacks necessary variables, such as no body weight, it is impossible to calculate the per-dose; in this case the answer should be directly classified as “insufficient information”—do not let the aggregator force a vote on incomplete information.

Second step, generate the first batch of 3 paths. Use a moderate temperature and require each path to produce an executable formula. Note that the aggregation step reads only the final structured answer from each path; lengthy reasoning text does not gain extra voting weight.

Third step, vote aggregation and validation. Suppose two paths give 20 mg and one gives 200 mg; the vote margin may seem sufficient, but the error cost of a dosage task is extremely high, so the vote margin alone cannot be the basis for decisions, and you still need to call the unit checker and range rules for verification. If 20 mg passes verification, output the answer with the calculation basis; if verification fails, do not continue replicating the same solution, but instead use different decomposition prompts to sample 2 more paths, changing the mechanism source of the paths.

Fourth step, stop. If 5 paths have been sampled and there is still no consensus that passes validation, refuse to answer automatically and escalate to an expert instead of continuing to vote. Throughout the process, the number of samples, the prompts used, normalization actions, validation results, and stop reasons are all written to the trace record, making this process auditable afterward.

The observables and failure actions of each stage can also be fixed: the sampling stage looks at the number of unique answers and the invalid rate, and on failure adjusts temperature or constrains the output format; the normalization stage looks at parse failures and unit conflicts, and on failure clarifies the input or refuses to answer; the aggregation stage looks at vote margin and effective sample size, and on failure increases mechanism diversity; the validation stage looks at the pass rate of rules and tools, and on failure escalates to humans instead of continuing to vote.

Reporting only “Self-consistency raises accuracy by 3%” cannot justify a deployment decision, because accuracy does not include cost. If this 3% is obtained at the cost of 40 times the inference volume, treating the gain as an algorithmic advantage misses the cost half of the information. Therefore evaluation must report single-sample performance, aggregate performance, and budget at the same time.

The baseline must include at least four strategies: greedy single-pass decoding, single sampling at the same temperature, majority voting with a fixed sample count, and a sequential strategy with a verifier. Greedy single-pass gives a lower bound without randomness; same-temperature single sampling separates “the fluctuation brought by temperature” from “the gain brought by voting”; fixed-sample majority voting is naive self-consistency; the sequential strategy with a verifier is the complete system. For each strategy, report task accuracy, average and P95 call counts, token cost, latency, and rejection rate, and plot a quality–cost curve so that accuracy always appears side by side with cost.

The evaluation set should also be sliced by dimensions: whether answers can be normalized, whether they are open-ended, whether a verifier is available, and question difficulty. Aggregated overall accuracy can mask differences across slices; for example, performance on open-ended tasks may be far worse than on tasks with normalizable answers. Also record each path’s final answer, normalization cluster, verification result, and stopping reason, so that a failure can be attributed to the generator, the aggregator, or the verifier, rather than being broadly classified as “the method is ineffective.”

Hyperparameter leakage must be eliminated: temperature, sample cap, and stopping boundary are selected only on the development set, frozen after selection, and the independent test set is reported only once. The key ablation experiment compares four path compositions under the condition that total token budget remains approximately unchanged—“more short paths,” “fewer long paths,” “paths from different models,” and “repeated paths from the same model.” It can answer the real question: does the gain come from computation amount, path diversity, or the verifier? The input of this evaluation is the four strategies and the frozen test set; the output is accuracy, P95 call count, token, latency, rejection rate, and quality–cost curve.

Self-consistency Decoding starts from the fragility of a single path and gradually builds an auditable decision chain, where each link addresses the question left by the previous link.

The starting point is this fact: sampling the same question again may produce a different answer. An early random branch choice can take the entire reasoning onto a different path, so a single path may happen to be wrong or happen to be right. Since the correct path appears more often across repeated samples, it is worth randomly sampling multiple paths for the same question so that voting has a chance to suppress accidental errors. This establishes the first link in the chain: sampling must preserve moderate diversity—too low a temperature will repeat the same path verbatim, while too high a temperature will produce many invalid reasoning paths.

With paths in place, we also need comparable votes. The final answer of each path is extracted, rather than the entire reasoning text being counted; normalization merges equivalent forms such as 1/2, 0.5, and 50% into the same cluster—too fine a granularity will split votes, and too coarse a granularity will merge votes. Then majority or weighted aggregation is performed to obtain a consensus answer.

But consensus is only a preference of the same sampling process, not a fact: shared training bias, the same erroneous formula, or prompt hints can all make errors highly correlated—high vote counts but low information. Correlated errors compress the nominal sample size into a smaller effective sample size, and voting gains saturate accordingly. Therefore, after aggregation, external verification must be introduced—computation, retrieval, testing, or independent rules—to verify the consensus with evidence that does not share the model's biases; high-impact conclusions cannot end with vote counts.

Cost and verification together determine when to stop: sample sequentially in batches and stop early when the vote margin reaches a freezing boundary, the verifier passes, or the marginal benefit falls below cost—rather than sampling a fixed forty times per question. Observations and decisions from each link are recorded, turning this chain from 'sampling a few more times' into a traceable, reproducible search process.