Post-training

At the end of pre-training, the model is already knowledgeable but cannot directly serve as an assistant. The reason lies in the quantity pre-training optimizes: text likelihood, that is, the relative probability the model assigns to the training text and its continuation. When likelihood is pushed up, it means the model has learned the common language regularities in the corpus—which words often appear next to each other, which expressions are more common, and how topics typically continue. However, likelihood does not encode intention judgments such as “what does the user want me to do now.” A model trained only to continue plausible text, when faced with a question, tends to complete a plausible passage rather than perform a task, follow a format, or refuse out-of-bounds requests.

Post-training is exactly what closes this gap. It swaps training data and feedback for forms closer to product goals—instruction demonstrations, preference comparisons, verifiable feedback—and thereby shapes several concrete capabilities: instruction following, answer style, tool use, safety boundaries, and reasoning strategy.

Conceptually, we need to distinguish two levels: post-training is the umbrella term for the whole stage, while fine-tuning is the concrete means inside the stage used to update parameters. Post-training’s input is a base model, plus instruction demonstrations, preference comparisons, or verifiable feedback; its output is an assistant model with updated parameters and the corresponding evaluation records. The mechanisms at each layer follow the same loop: generate candidate answers, compare candidates with targets, compute a loss or reward, then use it to update parameters. Each round of the loop changes the model’s distribution of answers when facing a certain prompt.

Interpret the result this way: what changes after the update is the probability that the model triggers the target behavior for a specific input—such as following instructions more reliably, answering in the agreed style, and appropriately refusing unsafe requests. It is not that the model suddenly acquires authoritative facts. Therefore, if basic capabilities, factuality, or a few language slices regress, even if the overall preference score rises, you cannot declare success on that basis. The capability boundary of post-training lies here: it adjusts the trigger probability of behavior, rather than injecting new factual knowledge.

To get a model that only knows how to continue text to learn to “answer as agreed when receiving an instruction,” the first step is to collect a large amount of high-quality “instruction–ideal response” demonstrations and continue training with cross-entropy. This is supervised fine-tuning (SFT).

Each training sample consists of two parts: the input is the instruction and optional context, and the target output is the token sequence in the demonstration response. The training method is the same as pre-training, reducing cross-entropy token by token: the model predicts the probability distribution of the next token based on the tokens it has already seen, and cross-entropy measures the gap between this predicted distribution and the true target—the true target is a one-hot distribution, where only the correct token has probability 1 and the rest are 0, so the loss at that position is −log p(target token | context), that is, the negative logarithm of the probability the model assigns to the correct token. Parameters are updated in the direction that increases this probability. As samples pass through again and again, the model keeps raising the probability that “this kind of instruction should be followed by this kind of answer,” and the behavior distribution changes accordingly.

What SFT establishes is a basic behavior distribution: when asked, answer rather than continuing arbitrarily; output in the agreed format; refuse when appropriate. After training, the format-compliance rate or refusal accuracy rises, and the correct explanation is that the model reproduces behaviors that appeared in the demonstrations more frequently, rather than that it already understands uncovered scenarios and can draw inferences about other cases.

The boundary of this mechanism comes directly from its dependence on demonstrations. What SFT learns is the demonstration distribution, and the coverage of the demonstrations is its capability ceiling: in scenarios that the demonstrations have not covered, the model may still fail. The quality of the demonstrations is also absorbed as is—when standards conflict with each other, the language distribution is unbalanced, or the entire text mechanically applies templates, the model learns all these defects into its parameters. Another boundary concerns knowledge: putting a large amount of the latest facts into SFT data makes them both difficult to update and difficult to trace, because after knowledge is compressed into weights, it is impossible to point out the source or replace it locally; constantly changing knowledge is better placed in retrieval-augmented generation (RAG), letting the model retrieve it dynamically when answering, rather than freezing it at training time.

When a question has no single right answer—when the “best answer” depends on user preferences, context, and style—SFT-style verbatim demonstrations are no longer sufficient. The second layer, preference optimization, uses a different signal: have annotators compare multiple responses to the same prompt, select the more preferred one, and then use RLHF or DPO to increase the probability of the preferred response.

Comparison signals are more flexible than verbatim reference answers: annotators do not need to write a perfect answer; they only need to judge which of two existing responses is better, which is closer to the real product judgment of “which type of reply users accept more.” But it is also only a proxy—it measures annotators’ relative preferences, not absolute correctness. If annotators habitually prefer longer, more confident, more user-pleasing responses, the model will optimize in that direction.

The inputs for this layer include the prompt x, the preferred response y⁺, the worse response y⁻, and a reference model that serves as a behavioral anchor. The output is still the updated policy model: it must widen the relative score between “good answers” and “bad answers,” while at the same time not completely discarding the model’s original language ability in order to cater to the preference data. This pair of “both must hold” requirements is written into a single objective:

J(θ) = R_preference(θ) − λ × KL(πθ ‖ π_reference)

Let’s read this expression step by step. θ is the model parameter being updated, and J is the total objective to be increased. R_preference measures how well the current model’s responses match the preference data—the more preferred a response is, the higher this term is. πθ is the current model’s response distribution, and π_reference is the reference model’s response distribution, that is, the distribution before updating or from the SFT stage. KL(πθ ‖ π_reference) measures how far the current distribution deviates from the reference distribution: the larger the deviation, the more likely the model is to lose the reference model’s original language ability. λ controls the trade-off between the two—the balance point between pursuing preference gain and not deviating too far. The total objective equals preference gain minus λ times the KL divergence, so training does not unconditionally pander to the preference signal but raises it under a constraint.

It should be noted that the expression above is a structural formulation for understanding RLHF-type objectives. DPO takes another path: it rewrites the comparison data “y⁺ is better than y⁻” into a classification form that can be directly optimized, without explicitly training a reward model of the same kind as in the above expression, but directly updating the policy.

How should the results be interpreted? An increase in preference win rate means only this: under the current annotators, prompt set, and candidate generation method, the target response is selected more frequently. It does not promise general quality improvement, nor does it promise an increase in factual accuracy. Preferences themselves vary across groups—one group of annotators’ preferences do not represent all users; the choice of reference model and the size of λ also change the final behavior. Therefore, during validation you must separately test sycophancy tendency, verbosity, factuality, refusal behavior, and cross-group performance; any regression in any of these indicates that the rise in preference scores has not translated into real improvement.

For tasks like mathematics and code that have objective answers, preference comparison is still only a subjective proxy—can we use stronger feedback? The third layer switches to verifiable feedback: unit tests, answer checkers, provers, or process verifiers. The input is the problem, the model-generated answer or intermediate steps, and the verification rule; the output is pass/fail, a score, or the specific step location where an error occurred.

Result reward and process reward are complementary in granularity. Result reward, which only looks at the final answer, is a sparse but well-defined signal—passing the test means passing, not passing means not passing, and there is no subjectivity about "which is better"; but it compresses all information into a single endpoint, so the model cannot tell which intermediate step went wrong. A process verifier gives feedback at each step, pointing out earlier where the deviation occurred, so the model knows where to make corrections; the cost is that process-level annotation is expensive, and the verifier may favor a particular solution method—a path that can pass its checks is not necessarily the only correct path.

What the model learns from this is not just "getting the answer right," but the whole loop of decomposing problems, searching paths, checking intermediate results, and correcting errors. This becomes an important training source for reasoning models: because each step has a decidable right or wrong, the model can self-generate a large number of trajectories, use the verifier to filter out erroneous trajectories, and then use the correct trajectories for training.

But a rising verification score must be interpreted with caution: it only shows that the model has become better at satisfying the current checker. The verifier is not the truth itself—the coverage of the test cases, the axioms adopted by the prover, and the way the rule code is written together determine what the "visible objective" is. A correct answer that is not covered will be judged wrong, and an incorrect answer that is covered may slip through. The weaker the verifier, the more easily it can be exploited by reward hacking: the model may learn to game the system—giving a solution that happens to pass the tests but is substantively wrong, exploiting random seeds, or pandering to the checker's quirks—with training reward rising while real capability stays unchanged. Therefore, hidden tests, adversarial examples, and independent final verification must be retained, and the divergence between training reward and real task metrics must be continuously monitored.

When post-training metrics improve, a follow-up question that is often skipped is: what does this improvement actually mean? The same score increase can correspond to three fundamentally different changes: the model has acquired a new capability it did not have before; an existing capability is triggered and invoked more reliably; or only superficial style such as output format and tone becomes more like the training samples. The three have very different actual value for the model, and conflating them will distort evaluation conclusions.

What post-training is good at changing is concentrated at the behavioral level. It can adjust the format and structure of responses, change the refusal boundary under ambiguous requests, influence tool selection when multiple available means exist, and train more systematic reasoning strategies, such as planning before solving. These are all questions of "when and how to invoke existing knowledge," belonging to the shaping of behavioral patterns.

Post-training is also good at redistributing the timing of invoking existing capabilities. A typical observation is: the base model can solve a certain problem with a limited number of samples, but it does not by default adopt this solution path; only one particular sample happens to take the correct route. Post-training can increase the probability that this strategy is triggered by default, turning correct answers from occasional into the norm. But the same mechanism has a downside: if the training data is too homogeneous, the model may prematurely apply a fixed template, mechanically applying it even in scenarios where that strategy is not needed. Therefore, "capability emergence" and "reliable capability invocation" must be measured separately. Evaluation should simultaneously look at pass@1 — the proportion of responses that are correct on the first attempt, the capability ceiling under more samples — whether the model can find the correct answer across multiple samples, and cross-prompt robustness — whether performance remains stable when the question is rephrased. Only by putting the three together can we determine whether an improvement actually added a capability, increased the accessibility of existing capability, or merely changed superficial style.

Some goals post-training does not guarantee to achieve. It cannot inject precise and updatable new facts into the weights, cannot fundamentally eliminate hallucination, nor can it guarantee stable generalization on out-of-distribution data. There is even risk of damage: training on overly narrow data can cause regression in other capabilities, or make the model excessively refuse to answer, blocking questions that could originally be answered.

Factual updates also involve two difficult problems: deletion and time. It is difficult to prove in the weights that an old policy has been completely removed — the model may oscillate between old and new facts, and cannot naturally provide a source for a piece of knowledge. When knowledge needs to be revocable and auditable, retrieval or tool queries are usually more appropriate than continued training: the former allows the data source to be replaced at any time, and the latter leaves a verifiable record, while knowledge trained into the weights is both difficult to delete precisely and difficult to attribute to a source.

A decrease in training loss alone is not enough to answer "whether the assistant is really better." Loss measures how well the model fits the training distribution; it cannot distinguish whether the model is learning abilities that users care about, memorizing examples, overfitting to format, or sacrificing dimensions other than helpfulness. Therefore, evaluation must be separated from training: on an independent held-out set, simultaneously measure helpfulness, factuality, safety, reasoning ability, output format, latency and cost, and examine slices by language, domain, risk, and length. A single overall score cannot mask a collapse in a single language or a regression on certain high-risk requests.

The most common mistake in the data loop is test set contamination. Newly discovered failure examples can be organized and fed back as training data, which is the value of the loop; but the test set must not be mixed directly into training, otherwise the model is like having seen the answers in an exam, and scores lose their meaning as a measure of ability. The correct approach is to keep a final evaluation set that never participates in training, and record the source and deduplication relationships for each batch of data, ensuring that the training, validation, and final evaluation sets are isolated from each other, so that every ability judgment is based on samples the model has never seen.

The timing of evaluation also needs to be designed, and its natural form is stage gates. After SFT, first check basic capabilities to confirm that the model has no obvious gaps in underlying skills such as formatting and instruction following, then proceed to the next stage. After preference optimization, check helpfulness and sycophantic tendencies, and whether refusal behavior is reasonable, because this stage is most likely to push the model toward "agreeing with everything." After verifiable training, check the performance on the target task itself and whether it still generalizes beyond the coverage of the verifier—because the model may have merely learned to please the verifier. If the overall score increases but certain user groups or task types regress, averaging cannot be used to cover it up; the purpose of the gate is precisely to catch such local regressions at each stage.

Viewed by stringing the three stages together, the essence of post-training is one and the same thing: data, objectives, and constraints act layer by layer on the same base distribution; each layer solves the problems left by the previous layer until the model's behavior distribution aligns with user intent.

Pre-training provides broad capabilities—language understanding, knowledge, and initial reasoning all form in this layer, but its training objective is only to predict the next word, not to serve user intent. A model that can continue text fluently may not follow instructions, nor necessarily refuse dangerous requests. It has the capability, but does not know when or in what form to use it.

SFT uses demonstration data to establish the basic behavior of an assistant, answering the question “what should an assistant usually do”: how to follow instructions, what format to respond in, and which requests to refuse. This layer teaches the behavioral skeleton; demonstrations tell the model a single correct answer.

But many requests have no unique answer. Two answers to the same question may both be fluent and both follow the format, yet clearly differ in quality: one is more complete, the other more perfunctory. Preference optimization deals with exactly this kind of quality trade-off where there is no unique answer; it uses comparison signals to tell the model which of two equally “valid” answers users usually prefer. The goal upgrades from “do as the demonstration does” to “choose, among available behaviors, the one users prefer more.”

Going further, for some tasks even “quality” is hard to judge by intuition, but there are verifiable criteria: whether the answer is correct can be checked, and intermediate steps can be verified. Verifiable feedback training teaches the model to search, check, and correct—after generating, actively verify, find errors, and then correct them, turning a one-shot answer into a solution process with self-checking.

The boundary of this chain is that when optimizing a proxy objective, the model may game the objective, learning to cater to metrics rather than real capability—for example, pleasing the verifier, applying fixed templates, instead of truly understanding. Therefore every layer needs constraints, and evaluations independent of the training objective are used to confirm that the model has genuinely improved, not just learned to perform on metrics. Ultimately, post-training shapes behavior: it determines when capabilities are invoked, in what form they are invoked, and which requests are refused, but it does not replace external factual systems, nor is it responsible for storing the latest, auditable world knowledge in the weights.

The same refund request can clearly show what each of the three stages changes. The user asks, “Can this order be refunded?” The continuation given by the same base model, the SFT assistant's response, and the response after going through preference optimization and verifiable feedback differ not in vocabulary but in what pressure each layer of supervision signals exerts. The three stages use different supervision signals: SFT imitates demonstrations, preference optimization ranks among feasible responses, and verifiable feedback rewards outcomes or processes that can be externally checked.

Figure 1 places the three candidate responses on the same pipeline for comparison:

Looking only at SFT likelihood, A scores the highest (0.45): it most resembles the common response style in training demonstrations and has the smoothest tone. But smoothness is not correctness — A asserts categorically that no refunds are possible after 30 days, ignoring the fact that quality issues may trigger an exception clause; it is “fluent but factually incomplete condition.” C has the lowest likelihood (0.20), and it overpromises “guaranteed full refund” without any order evidence to support it.

Preference probability raises B's win rate over A relative to the original candidate tendencies: B goes from 0.35 to 0.70, and A goes from 0.45 to 0.20. But “preference 0.70” is not factual accuracy — it only indicates that annotators show stronger approval of B's caution and responsibility, while whether B is actually correct still requires external rule verification to check: date, quality exception clause, order fields — none can be missing. The verification result has only B passing; A and C both fail, so B is retained. Here is a critical boundary: if the verifier only checks whether the answer contains the phrase “quality issue,” the model may learn to pile up keywords without verifying the order, which is reward hacking — the model optimizes the verifier's surface signal instead of real order verification. Whatever the verifier covers, the model tilts in that direction; this is a bias source that verifiable feedback training must guard against.

The real division of labor among the three stages is therefore clear: SFT first pulls effective behavior into the model's distribution, letting the model at least know that “answers should be organized like this”; preference optimization moves probability mass among candidates that already exist in the distribution, deciding which behavior is more often selected; verifiable feedback directs the search budget toward checkable objectives, letting the model learn to check and correct. No stage can exceed the coverage of its data, rewards, and evaluation — behaviors not covered by data will not appear out of thin air, quality dimensions not covered by rewards will not automatically improve, and degradation not covered by evaluation will not be detected.

The failure boundary is also worth remembering: post-training changes how existing capabilities are accessed, and it may produce over-refusal, sycophancy, templated phrasing, and domain regression. Go-live decisions must retain three perspectives at once — base model capability, target behavior, and safety slice — not just look at a single total reward.

Post-training, fine-tuning, alignment, RLHF, and fact updating are often mixed up, but they sit at different levels. Fine-tuning broadly refers to continuing gradient updates on pre-trained weights; post-training is one of its stages. Alignment is one of the goals of post-training, namely making model behavior align with user intent and safety boundaries. RLHF is one concrete technical route for preference optimization. Fact updating is a separate, independent need, and it is not the same thing as behavior shaping. Only by laying out the levels can we avoid mistaking “switching training methods” for “injecting a new piece of knowledge.”

Around the core concepts on this page, the dependency relationships can be layered as follows. The prerequisite concepts are pre-training, supervised fine-tuning, cross-entropy, and sampling: without first understanding the source of the base model’s capabilities and the basic operation of fitting demonstration distributions with cross-entropy, it is impossible to understand what each subsequent step is doing. The core of this page consists of SFT, preference data, preference-optimization routes such as RLHF and DPO, verifiable feedback, and KL constraint and regression: the KL constraint prevents the model from drifting too far when optimizing for reward, while regression serves as a reminder that every step may sacrifice previous capabilities. The immediate extensions include alignment, reasoning models, test-time compute, and reward hacking: the models produced by post-training are exactly the objects discussed by these topics, and reward hacking is the most direct risk extension. Further outward are engineering extensions: model evaluation, training data governance, RAG, human-in-the-loop, and safety guardrails—these are the system-level supporting elements for post-training.

The evaluation roadmap unfolds in at least three layers: first, look at whether the offline objective is optimized—that is, whether the training signal itself decreases or improves; second, look at whether the target behavior improves—that is, whether the user-perceivable performance truly gets better; finally, look at whether capabilities that were not directly optimized regress, because post-training often exacts a cost on dimensions that are not visible. Beyond these three layers, a cost-benefit comparison is also necessary: compare with prompt engineering, RAG, or a stronger base model at equal cost, because post-training is not always the optimal solution. If the requirement is merely to make the model know the latest policy, writing facts into the weights is typically worse than retrieval—retrieval can update data sources at any time, delete old entries, and trace back to the source, whereas facts in the weights are hard to update individually, hard to prove deleted, and hard to explain their source.