Chain of Thought (CoT)

Faced with problems requiring multi-step computation, the weakness language models most readily expose is answering without thinking: they tend to give the final answer directly in one step, cramming all the intermediate calculations that should have been broken out into a single implicit "mental arithmetic". The consequence of a one-step mistake is easy to see — the answer is wrong, but the user cannot see which step went wrong.

Chain of Thought targets exactly this weakness, and the method is surprisingly simple: do not let the model jump directly to a conclusion, but require it to first write out the reasoning process step by step, and only then give the final answer. This is like a teacher asking you to "show your work" on an exam instead of just writing down the final number. After writing out the steps, each reasoning step is laid out in the open, becoming a local state that can be checked individually.

A concrete example shows the difference. For the same problem that requires "first adding, then dividing," when the same model answers directly, it gives 6, which is wrong; as soon as it is asked to write out the intermediate process of "first adding, then dividing," the final answer becomes 7, which is correct. The difference is not in the model but in whether it is required to expand the reasoning into words.

Figure 1 shows exactly this contrast: for the same problem and the same model, answering directly tends to fail in implicit mental arithmetic; writing out the reasoning step by step is equivalent to breaking a difficult problem into several simple operations, each step is less likely to go wrong, and accuracy therefore clearly increases.

From this we can derive the definition of Chain of Thought: it makes the model first write out step-by-step reasoning and only then give the conclusion, used to solve the problem of models answering without thinking and getting multi-step problems wrong in one step. Its input is a multi-step problem plus an instruction or example requiring step-by-step answering; what it changes is the model's computational path — replacing "directly guessing the conclusion" with "writing out checkable local states step by step, and then deriving the answer from them"; its output contains both intermediate steps and the final answer.

We should also see its limits: writing out steps usually means the difficult problem is broken into smaller steps that are easier to get right, but the intermediate text itself can still be wrong, or it may be just a post-hoc rationalization of the conclusion. Chain of Thought increases the chance that reasoning is seen and checked, but does not guarantee that every step is truly correct.

Why does writing it out make such a difference? Isn’t the model already capable of reasoning? To answer this question, we first need to recall the basic way large language models generate text: one token at a time, with no global scratchpad throughout the process (see the “Large Language Models” deep-dive page). This fact determines the difference between the two answering approaches.

Answering directly means asking the model to jump to the final answer of the entire hard problem in one step. All the intermediate multi-step reasoning has to be completed implicitly in a single generation: the model can do some “mental arithmetic” internally, but it has no opportunity to lay out intermediate results for checking. If any step in the multi-step reasoning quietly goes wrong, the final answer is wrong, with no room for correction.

Writing out the reasoning is equivalent to breaking a hard problem into a sequence of small steps. Each step only needs to “get the next step right” based on the previous steps: the task is not to solve the whole problem in one go, but to advance one small step from the given local state, so the per-step probability of error is naturally lower. More importantly, the steps already written become context for subsequent steps—the model can “look at what it just wrote” and continue forward, rather than recomputing from memory, so it is much more stable.

In one sentence: Chain of Thought can be compared to giving the model an explicit scratchpad—adding intermediate tokens so that later tokens can be conditioned on the steps already generated. Note that this is a useful intuition, not equivalent to these words fully revealing the model’s actual internal computation.

The value of step-by-step calculation lies in the checkability of the steps. Taking “23 people, then 5 more arrive, divided equally into 4 groups” as an example, it can be written as a sequence of states: initially n₀ = 23; after 5 more arrive, n₁ = 23 + 5 = 28; after dividing into 4 groups, each group g = n₁ ÷ 4 = 7. Finally, we can reverse-check 7 × 4 = 28 to confirm that the grouping result is consistent with the total number of people. If an intermediate step were mistakenly written as 23 + 5 = 27, the user could see at a glance that the error is in the second step rather than the last—the location of the error is exposed by the steps. But it must also be admitted that the language steps themselves can still be written incorrectly: the error of miscounting 23 + 5 as 27 does not automatically disappear just because it is written out. Therefore, any arithmetic that can be precisely calculated by a machine is best handed over to a calculator or code for verification, rather than relying on mental arithmetic in language steps.

Putting it all together: the reason Chain of Thought works can be understood as providing the model with an explicit scratchpad, answering the question of “why writing it out is more accurate than not writing it.” Its starting point is the fact that “text is generated word by word, with no global scratchpad”; the approach is to first break a hard problem that would be answered in one step into a sequence of small steps, then make the already-written steps the context for subsequent generation; the output is more stable step-by-step reasoning. The checkable intermediate steps make the location of errors visible, but the language steps themselves can still be written incorrectly—whatever can be calculated is best handed over to a calculator or code for verification.

After knowing that chain of thought is useful, the next question is operational: how do we get the model to actually write out the intermediate steps instead of continuing to jump straight to the answer? There are two common approaches, and the only difference is whether examples are provided.

The first is few-shot chain-of-thought: put one or two “with complete reasoning process” examples in the prompt, first demonstrate the pattern of “problem → step-by-step reasoning → answer”, then give the actual problem to solve. Upon seeing the examples, the model will follow the format of “reason first, then answer.” This ability to change behavior through examples is in-context learning—the model is not retrained; it just learns and applies from the demonstrations in the prompt on the fly (see the “In-context Learning” deep-dive page).

The second is zero-shot: no examples at all; just add a sentence after the question, “Let's think step by step” (Let's think step by step), which is often enough to trigger the model to write out steps. It does not require carefully prepared examples, has low cost, and its effectiveness depends on how familiar the model itself is with the “step-by-step solution” format.

Whichever way, it is essentially a technique in prompt engineering: chain of thought is the representative of the prompting pattern “make it step-by-step.” It can be triggered by examples precisely because in-context learning provides the fallback—when you demonstrate in the examples that “the answer should include reasoning,” the model follows suit. So these two concepts are two sides of the same mechanism: chain of thought describes the desired behavior, and in-context learning describes why the model follows the demonstration in the examples (see “Prompt Engineering” and “In-context Learning”).

To sum up the triggering methods: the problem it aims to solve is how to reliably elicit intermediate steps; its inputs are the task itself, model capability, number of examples, and output requirements; the approach is to first give one or two examples with steps for the model to imitate, or simply to ask it to process step by step; the output may be either a direct answer or a step-by-step response, depending on whether the trigger succeeds. It should be noted that being triggerable by examples only shows that the model is doing in-context learning, not that every step is reliable—high-risk results still need to be handed over to calculators, code, or source verification.

A single chain of thought occasionally makes a mistake at one step, and the entire chain then follows that mistake all the way to the end. To be more reliable, the direction is clear: have the model think multiple times, and then use some rule to combine the multiple attempts. Two representative variants correspond respectively to the two approaches of “vertical diversity” and “horizontal expansion”.

Self-consistency takes the vertical route: for the same problem, have the model solve it independently multiple times. To make the multiple solutions differ from each other and cover more possible lines of thought, use a higher temperature during sampling to create diversity; then tally the final answers from each chain and take a majority vote as the conclusion. A majority vote naturally offsets occasional errors in individual chains, so it is more stable than trusting a single chain (see “self-consistency”).

Tree of Thoughts (ToT) takes the horizontal route: like self-consistency, it compensates for the shortcomings of a single chain by “generating several candidates and then combining and selecting the best.” The only difference is how the candidates are organized—not keeping multiple full paths until the end and then voting, but organizing candidate thoughts into a tree structure and choosing among them through comparison. Both belong to the “think multiple times and then combine” approach, just more structured (see “Tree of Thoughts (ToT)” for details).

The common point of the two variants is that both elaborate on “having the model think more and then merge”: if a single chain is too thin, bring in several more and vote; if one direction is too narrow, put multiple candidates side by side to compare and select the best. In addition to the single-chain result, their inputs include a sampling budget and rules for merging candidates; self-consistency votes on the final answers from multiple independent samples, while Tree of Thoughts organizes multiple candidates into a tree structure and then chooses among them; the former outputs the answer after voting, and the latter outputs the chosen line of thought and conclusion.

The costs and boundaries should also be seen clearly: these methods trade more computation for more stable answers. Multiple paths arriving at the same conclusion usually indicates that the answer is more trustworthy, but if the candidate chains are highly correlated—essentially repetitions of the same error—then “thinking more” may still be consistently wrong, just wrong with more confidence.

In the last couple of years, a specialized class of “Reasoning Models” has emerged. They are also known for “thinking more and computing longer,” which can easily lead people to think they are just an enhanced version of Chain of Thought. The two are indeed related, but they differ in origin and form.

Chain of Thought was originally a prompting technique: the capability is in the model, and it is elicited from the outside through instructions or examples. Without any additional training, just by changing the way you ask, the model begins to write intermediate steps.

Reasoning Models take a different path: through specialized post-training, the model learns how to use more reasoning budget. The so-called budget refers to the extra computation allowed to be spent—it may take the form of longer reasoning trajectories, or it may take the form of generating multiple candidates, performing search, calling verifiers or tools. Moreover, these raw intermediate trajectories are not necessarily shown to the user; what the user sees may be only the condensed final answer.

Therefore, the two cannot be equated. The comparison can be viewed along four dimensions: where the capability comes from—elicited through prompting, or learned through post-training; how the budget is controlled—temporarily set by the questioner, or allocated by the model itself according to the task; what the computation looks like—writing steps linearly, or being able to search, generate multiple candidates, and call tools; whether the trajectory is visible—the intermediate steps of Chain of Thought are usually written in the answer, whereas the raw trajectory of Reasoning Models can be hidden. The conclusion of the comparison is: both use more reasoning, so they are related; but “writing out a long process” is not the definition of Reasoning Models, and hiding the intermediate process is entirely consistent with the practice of Reasoning Models.

Intermediate steps are not free. Every extra generated step consumes real resources, so before using it, it's worth thinking clearly: when does this “scratch paper” actually become more trouble than it's worth?

The most direct cost is that it is slower and more expensive. Writing out the reasoning means the output becomes longer: for the same problem, the model has to generate an extra batch of tokens, so users wait longer, and in token-billed scenarios the cost rises proportionally. Second is window occupancy: long reasoning squeezes the precious context window budget; the part of the window consumed by the reasoning process can no longer be used to hold more background material or conversation history (see Context Window). The third cost is the easiest to overlook: not every problem needs it. Simple questions can already be answered correctly in one step; forcing chain of thought only makes the answer wordy and also wastes those first two costs for nothing. The value of chain of thought is concentrated in genuinely multi-step reasoning problems—there the benefit is enough to cover the cost.

There is also a more subtle reminder: the reasoning a model “writes out” is not always equal to its “actual internal computation.” Sometimes the steps look plausible, but the conclusion is wrong; sometimes the conclusion happens to be correct, but the steps were fabricated after the fact. What chain of thought improves is accuracy, not a guarantee of correctness; likewise, these words should not be taken as a reliable explanation of the model's internals.

Putting the costs and trade-offs together gives a pre-use decision checklist: first judge whether the problem is truly multi-step and whether the result can be externally verified; then see whether the latency, cost, context budget, and error risk are acceptable; only then choose among “answering directly,” “generating step by step,” and “handing off to a tool for calculation.” Longer output usually means slower and more expensive, and it also squeezes the window; forcing step-by-step on simple questions only makes things wordy. To sum up: chain of thought neither guarantees correctness nor is reliable evidence of interpretability—it is a useful but costly piece of scratch paper.

Connecting the previous sections, the complete causal chain of chain of thought is as follows.

The model generates only one token at a time, and the entire generation process has no global draft. Under this premise, having the model answer directly is equivalent to requiring it to produce the answer to an entire hard problem in a single leap, with all intermediate reasoning carried out invisibly and no chance to correct mistakes. What chain of thought does is replace “answering a hard problem in one step” with “proceeding step by step”: write the reasoning first, then give the conclusion.

Why is writing out the steps more accurate? Because after breaking it into small steps, each step only needs to correctly append the next small step on the basis of the previously written steps, making each individual step easier to get right; moreover, the steps already written become context for subsequent generation, so the model reasons by looking at its own draft rather than recomputing from scratch.

To trigger it, you can provide one or two examples with reasoning processes—this relies on in-context learning, where the model learns from the examples on the spot what “reason first, then answer” looks like; or you can give no examples at all and simply add “Let’s think step by step.”

When a single chain is not stable enough, you can strengthen it: self-consistency solves the same problem independently multiple times and then takes a majority vote on the answers; tree of thought does not follow a single path but expands multiple branches, evaluates, backtracks, and selects the best like a search.

Zooming out, dedicated reasoning models are related to chain of thought but not identical. Reasoning models learn through post-training how to use more reasoning budget—budget can be spent on longer trajectories, or on multiple candidates, search, verifiers, or tools, and the raw trajectories may not be shown to the user. It is not a “trained version” of chain of thought, but another path to acquiring reasoning ability.

Finally, the cost: writing out reasoning means the output gets longer, slower, and more expensive, and it also takes up the context window; at the same time, the steps themselves are not guaranteed to be correct, and simple problems are not worth using it for.

Only by being able to explain this chain from beginning to end can you truly grasp the core of chain of thought: use “the model generates token by token and has no draft” to explain why writing out reasoning is more accurate, and clearly explain the difference between chain of thought and reasoning models.

Chain of Thought itself is not complicated, but to truly understand it, you need to first stand on several foundational concepts: how large language models generate token by token, the entire process has no global scratchpad—this is the entire explanatory source of "why writing it out is more accurate"; in-context learning explains why the model will follow along when given one or two examples; prompt engineering provides the concrete techniques for triggering it. Without any one of these, Chain of Thought is just a slogan like "writing more steps is more accurate".

The core of this page itself lies in four things: writing out reasoning, turning implicit mental calculation into checkable intermediate steps; breaking down hard problems, cutting a multi-step problem into a string of small steps; triggering methods, using few-shot examples or a one-sentence step-by-step instruction to elicit steps; cost, slower, more expensive, takes up the context window, and does not guarantee correctness.

Following this thread, we can continue to extend further. Adjacent to this chapter are its strengthened version and close relatives: self-consistency and Tree of Thoughts (ToT) reinforce a single chain in the two directions of "casting more votes" and "expanding into a tree", respectively; reasoning models are what reasoning ability looks like after moving it from the prompt level to the training level; the context window is the prerequisite concept for measuring the space cost of "scratch paper". Going further out, ReAct alternates reasoning and acting, reflection lets the model examine the process it just wrote down, and evaluation is the final measure for testing whether all of this is truly effective.