RLHF and Preference Alignment: Turning “What People Prefer” into a Trainable Signal

Most open-ended tasks do not have a single unique correct answer. For requests such as “explain recursion,” “help me draft a refund email,” and “summarize this code,” a model can give countless plausible responses, and it is difficult to use a single “standard answer” to determine which one is correct. But when humans are faced with two candidate responses, they can usually compare which one is better: more accurate, more useful, more truthful, or more harmless. This intuition is exactly the starting point of RLHF—not to read out a hidden “full-score answer” to the model, but to turn the relative judgment of “which answer do people prefer?” into a trainable signal.

Supervised fine-tuning (SFT) takes another path: it requires demonstrators to write out the ideal response by hand, and then uses these demonstrations to train the model directly. This path has two practical bottlenecks. First, cost is high: writing high-quality, comprehensive demonstrations requires a great deal of expert time. Second, coverage is limited: the scenarios demonstrators can think of are always finite, and when encountering new requests outside the distribution, the model may have no “standard answer” to refer to. Preference learning reformulates the problem as: given response A and response B, which better meets requirements such as helpfulness, truthfulness, and harmlessness? The model no longer needs to know absolute scores; it only needs to learn an optimizable direction from massive relative comparisons.

Take a concrete example. Faced with “explain recursion,” response A is conceptually completely accurate but full of jargon, so beginners find it hard to follow; response B is equally accurate and includes a minimal code example to aid understanding. The annotator chooses B, which provides a relative preference—B is more suitable for this scenario than A—rather than declaring “what is the objective full-score value of recursion?” The key to RLHF lies in compressing this kind of comparative choice, turning the information “B is better” into a gradient signal that tells the model which direction to adjust, rather than trying to read out an absolute truth that does not exist.

The overall idea of a classic RLHF pipeline is to proceed in three steps: first teach the model basic behavior, then let the model learn human preferences, and finally move the policy toward high reward. These four steps can be seen as a processing chain from 'being able to speak' to 'speaking in a way that satisfies people.' [Translator’s note: The source describes three stages but then refers to four steps; the detailed pipeline enumerates four steps.]

The first step is supervised fine-tuning (SFT). The base model has only learned to predict text and does not necessarily know how to 'follow instructions.' Train it with human demonstrations or high-quality instruction data so that it learns to understand a prompt and give an answer that meets the requirements. This step addresses basic behavior: the model begins to treat the input as a task rather than simply continuing the text.

The second step is sampling and comparison. For the same prompt, let the current model generate multiple candidate responses, and then have people rank them or make pairwise choices. This step does not require annotators to write ideal answers; it only asks them to judge 'which one is better.' These relative judgments are the source of the preference signal to be learned later.

The third step is reward modeling. Use the comparison data collected above to train a reward model, which outputs a relative score for the 'prompt + response' pair. What the reward model needs to learn is not absolute truth but a computable approximation of 'this response is generally more preferred under human comparison.' The order of scores should match human ranking as closely as possible.

The fourth step is policy optimization. With the scores given by the reward model as the target, use reinforcement learning methods such as PPO to increase the model's expected reward; at the same time, constrain the policy during optimization from deviating too far from the reference model, so as not to degenerate into strange behavior in order to chase high scores. This step turns 'what people like' into a direction the model can actually move toward.

It should be noted that these four steps are the classic InstructGPT route; it provides a general framework, not the only standard answer. Modern alignment systems do not necessarily use the same optimizer or the same annotation method; some methods omit independent reward modeling, and some use different forms of preference data, but most still follow the general order of 'basic behavior first, then preferences, then optimization.'

The reward model needs to learn not the absolute value of an answer, but the relative win rate between two answers. Annotators give not “how many points this answer is worth” but “which of the two answers I prefer.” Therefore, the reward model’s goal is to make “the preferred answer’s score is higher than the rejected answer’s score” hold as often as possible.

If annotators prefer answer y_w over y_l, this preference can be written in Bradley–Terry form as a probability: given prompt x, the probability that answer y_w beats answer y_l equals the value obtained by applying sigmoid to the reward difference between the two answers.

P(y_w ≻ y_l | x) = σ( r(x, y_w) − r(x, y_l) )

Here σ is the sigmoid function, which compresses any real number into the range 0 to 1; r(x, y) is the score that the reward model gives to the “prompt + answer” pair. The meaning of this formula is straightforward: the larger the reward difference between the two answers, the closer the preferred answer’s win probability is to 1; the closer the reward difference is to 0, the closer the probability is to 0.5, meaning the two are almost indistinguishable. Conversely, if the rejected answer’s score turns out to be higher instead, the probability drops below 0.5, indicating that the model’s judgment contradicts the human choice.

When training the reward model, the goal is to increase the score difference between the preferred answer and the rejected answer. When the model sees a comparison of “A is better than B,” it adjusts its parameters to make r(x, A) − r(x, B) larger, thereby making P(A ≻ B | x) close to 1. What is optimized here is the relative ordering, not some absolute “quality score.”

For this reason, reward scores are meaningful only within the current data, annotation guidelines, and candidate distribution. They are a relative ruler: a score of 8 points only means “within this candidate set and according to this group of annotators' guidelines, it usually ranks ahead,” and it cannot be interpreted as a fixed “quality score of 8” in the real world. Change the annotators, change the guidelines, or change the candidate distribution, and the same answer may receive a completely different score.

If the goal of policy optimization were only to push reward model scores higher, the policy would quickly learn to exploit this proxy metric. After all, the reward model is only an approximation; it can also give high scores to certain loophole-style answers—repetitively piling on pleasing wording, overconfidence, pandering to surface formatting. Once the model discovers that such answers can reliably obtain high rewards, it will go further and further down this path, eventually drifting away from the reference model that originally speaks like a human.

To suppress this drift, the policy optimization objective function adds a constraint term in addition to reward. What is maximized is the expected reward of the answer, minus β times the KL divergence between the current policy and the reference policy:

max π_θ 𝔼[ r(x, y) ] − β · KL( π_θ(· | x) ‖ π_ref(· | x) )

This expression consists of two terms. The first term 𝔼[ r(x, y) ] encourages the model to generate high-reward answers and is the direct driving force of optimization. The second term KL( π_θ ‖ π_ref ) measures the distance between the output distributions of the current policy π_θ and the reference policy π_ref for a given prompt x, and is deducted from the objective as a penalty. The coefficient β controls the trade-off between the two: the larger β is, the less the model dares to deviate from the reference policy, and the more conservative the update; the smaller β is, the stronger the dominance of the reward score, and the more aggressively the model can pursue high scores.

The role of the KL constraint is to make updates more conservative, thereby reducing the risks of language degeneration, mode collapse, and reward over-optimization. Language degeneration refers to the model beginning to repeat bizarre wording in order to chase scores; mode collapse refers to output diversity being sucked away by a few “high-score routines”; reward over-optimization refers to reward scores continuing to rise while true quality actually worsens. The KL term adds a speed bump to these phenomena.

But the KL constraint itself is not a safety proof. It only keeps the policy from going too far at once; it does not guarantee that the direction it goes is correct. A key pitfall is: “reward increase” only means that the output better matches the reward model's judgment, not automatically that it is more truthful, safer, or more aligned with all users' expectations. The error of the reward model itself lies between reward scores and true quality; KL can only limit the magnitude of drift and cannot replace independent evaluation.

Treating human preferences as a training signal does not mean obtaining an objective, unbiased yardstick. Annotators can only judge based on the responses in front of them and the guidelines at hand, and this judgment is affected by factors such as knowledge background, cultural habits, wording style, and time pressure. The so-called “what people prefer” is itself a noisy proxy, not pure ground truth.

Noise first shows up as systematic bias. Annotators often favor longer responses, because length tends to give the impression of being “more careful and more complete”; favor more confident responses, even when the confidence rests on errors; and also favor responses with prettier formatting, even if the content has no substantive difference. These preferences are not always related to true quality, yet they steadily seep into the comparison data.

Second, the limits of annotators’ abilities can create errors. When faced with specialized factual questions, annotators often cannot verify correctness on the spot, so they may mistake “sounds professional” for “factually correct.” Once such misjudgments enter the training data, they are learned by the reward model as valid signals.

More fundamentally, different annotators assign different weights to “helpful” and “harmless.” Some think providing more information matters most, while others think avoiding any potential harm takes priority over satisfying the request. Two equally reasonable judgments may produce opposite comparison results for the same pair of responses. Forcing such disagreements into a single ranking itself loses information.

In engineering, addressing this noise relies not on eliminating disagreements, but on making the noise explicit and controlling it: clarify annotation guidelines (rubric) so that judgment criteria are actionable; train annotators to narrow differences in understanding; measure annotator agreement to quantify the reliability of the data; preserve disagreements and do not forcibly flatten genuine value conflicts; and involve experts in judgment in high-risk domains. Preference data obtained this way still has noise, but the source, scope, and impact of the noise are made explicit as much as possible, rather than being treated by default as clean truth.

Direct Preference Optimization (DPO) and classic RLHF use the same kind of preference data—both are pairwise comparisons of the kind “response A is better than response B”, but DPO bypasses an explicit reward model and an online reinforcement learning loop. The classic route first trains a reward model, then uses algorithms such as PPO to optimize the policy in a reinforcement learning loop; DPO instead rewrites the preference objective directly as the log-probability difference between the policy and a reference policy, thereby directly increasing the relative probability of the preferred response. The intermediate standalone reward model is omitted.

The benefit of this approach is that it is usually easier to train. A reinforcement learning loop involves repeated interactions of sampling, reward scoring, and gradient updates, which is complex to engineer and sensitive to hyperparameters. DPO falls back to an offline form more like ordinary supervised learning, with more stable training and simpler implementation, and is friendlier for scenarios that require stable, simple offline training.

But DPO does not thereby escape the common dependencies of preference alignment. It still requires preference data, a reference model, and appropriate hyperparameters, and it may still inherit biases in the data—annotator bias and limitations of the candidate distribution do not automatically disappear just because one reward model is removed.

Therefore, the accurate statement is: DPO is a preference alignment method, not a synonym for the classic “reward model + PPO” pipeline. The two share the goal of turning relative preferences into an optimization direction, but their implementation paths differ. A practical selection tip is: when the goal is stable, simple offline training, consider DPO first; when you need online exploration, combining complex rewards, or interaction with an environment, the reinforcement learning route still retains its value, because an explicit reward model and online sampling can support a more flexible objective structure.

Whether alignment actually improves the product cannot be concluded from reward scores during training, because the training reward is itself the object being optimized. Validation must rely on independent evaluation, and must be rechecked on tasks and risk sets that did not participate in training—those scenarios the model has never seen are where generalization ability is tested.

A complete validation chain is roughly as follows: first clarify the goals and conflicts, understanding where “helpful” and “harmless” pull against each other; then design annotation guidelines to make the evaluation criteria actionable; next check annotation consistency to confirm the data itself is reliable; then train the preference objective and continuously monitor changes in reward and KL; afterwards use independent capability and safety evaluations to recheck on held-out sets; finally provide a backstop through red-team attacks and continuous monitoring after launch.

During evaluation, you need to observe multiple dimensions at the same time rather than looking only at a single metric: task success rate measures whether the model accomplished what it was supposed to do; factuality measures whether the answer is truthful; safety violations measure whether harmful output is suppressed; over-refusal measures whether safety constraints have been overcorrected, causing questions that should normally be answered to be rejected outright; group differences measure whether systematic bias exists among different user groups; and human blind evaluation provides the final human judgment from the same source as the preference data.

If a report provides only the reward model score, it cannot rule out two typical failures: one is reward hacking, where the model has learned to exploit loopholes in the reward model, so the score is inflated while the actual quality has not changed; the other is overfitting to a single evaluator, where the model specifically caters to the scoring habits of one automated evaluator, and its true colors are revealed as soon as a different evaluator is used. Only by viewing reward scores within a multi-dimensional, independent, and held-out evaluation system together can we judge whether alignment has actually improved the product.

There are several widely circulated claims about RLHF that need to be corrected.

First, RLHF is not a one-time proof of values, but an engineering method for compressing a set of preferences into the model. What it does is turn the relative preferences under specific annotators, specific criteria, and a specific candidate distribution into a trainable proxy signal, rather than starting from some external truth and "hard-coding" values into the model once and for all. Preference data has noise, the reward model has error, and the optimization process has limitations; all of these determine that it can only be an iterative approximation engineering process, not a once-and-for-all proof.

Second, RLHF is not equivalent to "making the model never make mistakes". It optimizes only a proxy signal on limited data, and there is no equal sign between the proxy signal and "being true, safe, and correct for all users". Increases in reward scores can be contaminated by reward hacking or data bias, and new errors in the real world can still appear.

Third, SFT, reward modeling, and policy optimization have different roles; you cannot loosely call the entire process "reward model". Supervised fine-tuning is responsible for teaching the model to answer according to instructions, reward modeling is responsible for compressing human comparisons into scores, and policy optimization is responsible for moving the model in the direction of higher reward. The three are different stages in the pipeline, each with its own inputs and outputs; conflating them will obscure their respective failure modes.

Fourth, the relationship between alignment and capability is not fixed. The two can promote each other—clearer answers are both more helpful and easier to evaluate correctly; they can also conflict—overly pursuing safety can suppress helpfulness, or overly pursuing helpfulness can relax safety. Therefore we cannot draw conclusions from a single dimension; we need to evaluate by dimensions, measuring capability, factuality, safety, and so on separately, and then judge whether the overall trade-offs are reasonable.

Let’s use a refund scenario to tie together the previous steps. Suppose the model generated three candidate responses to the same refund request: A directly guarantees a refund, B verifies the order and then explains a quality exception, and C refuses in all cases. The annotators chose B among these three, on the grounds that it both verifies the order and gives a clear explanation of the quality exception.

The first thing preference data does is turn these candidates from “three parallel responses” into a set of pairwise orderings, such as B ≻ A and B ≻ C. Then the reward model compresses each response into a scalar score, translating the pairwise comparisons into one-dimensional numerical values. Suppose the reward model gives r(B) = 1.2, r(A) = −0.6, and r(C) = −0.1; then the win probability for each comparison can be computed.

For B versus A, the reward difference Δr = 1.2 − (−0.6) = 1.8; substituting into the sigmoid gives σ(1.8) ≈ 0.858, meaning the model predicts an 85.8% probability that B beats A. For B versus C, Δr = 1.2 − (−0.1) = 1.3, and σ(1.3) ≈ 0.786, meaning the model predicts a 78.6% probability that B beats C. For A versus C, Δr = −0.6 − (−0.1) = −0.5, and σ(−0.5) ≈ 0.378, meaning the probability that A beats C is only 37.8%; the model leans toward C, but this does not mean C is absolutely better, only that it is preferred relative to A.

This example exposes two properties of scalar rewards. First, the zero point of the reward has no absolute meaning: adding 100 to all rewards at the same time leaves the differences Δr and all win probabilities completely unchanged, showing that scores are meaningful only in relative comparisons. Second, compressing multiple dimensions such as truthfulness, helpfulness, and appropriateness of refusal into a single one-dimensional r discards the structure between dimensions. Compression makes optimization convenient, but it also hides preference conflicts and minority opinions. Therefore, training should retain per-dimension evaluation to avoid the model learning only superficial features that most easily please annotators.

This set of preference scores next drives the policy update: responses with high win probabilities like B receive more weight, and the policy moves in the direction of producing such responses more often, subject to the KL constraint. The whole process can be summarized as: human pairwise comparisons first provide an ordering, the reward model then compresses the ordering into a scalar, and constrained policy optimization then moves the model toward higher scores.

Also be wary of a failure boundary: these candidates were sampled from the old policy, and the preference model has only seen this distribution. When the new policy starts generating out-of-distribution text in order to chase higher scores, the reward model is extrapolating when it assigns scores, and the risk of distortion rises significantly. The KL constraint, online spot checks, reward model ensembling, and independent final validation can together limit this risk, but none of them can prove that the system has no vulnerabilities.