AI Alignment

Alignment addresses the question: when a model is already strong enough to perform a variety of tasks, what exactly should we 'align' it to? The answer is: to make the model's behavior conform to the true intentions and values of humans. The key phrase here is 'true intentions'—not a casual remark from a user, nor the literal surface meaning of a single input, but the outcome that a person truly wants after deliberation in the full context, and the value judgments they are willing to uphold over the long term.

The industry usually summarizes this intention into three parallel goals:

• Helpful: truly helping, producing results that have practical value for the user's current task.
• Honest: no fabrication or exaggeration; when uncertain, clearly say so; do not state content without basis as if it were certain.
• Harmless: do not assist in causing harm, do not actively amplify risks, and do not bypass safety boundaries to complete a task.

Alignment is not about picking one of these three to maximize, but about making the model perform appropriately on all three at the same time. There is real tension among the three goals: a model can fabricate details in order to be 'helpful', or refuse a request it should have answered normally in order to be 'harmless'. As a complete set of techniques and goals, alignment addresses the difficulty of satisfying them simultaneously.

Viewed at the system level, alignment is an ongoing process, not a value that can be computed once. Its inputs include the task itself, stakeholders' preferences, relevant policy norms, and the model's own capability boundaries; its outputs are changes in behavior that can be evaluated in a given scenario—for example, whether the model produces less harmful content on similar questions and whether it is less likely to confidently give wrong answers. This means the effect of alignment must be reflected in observable and measurable behavior, not an abstract 'value score'.

At the same time, two boundaries need to be drawn clearly. First, alignment does not mean the model always obeys any user: if a user's request conflicts with honesty or harmlessness, an aligned model should exercise restraint, clarify, or refuse, rather than comply without principle. Second, alignment does not mean instilling a static set of dogma into the model, because intentions, norms, and risk boundaries themselves change with context and over time. Alignment is therefore always an ongoing process of 'making behavior more aligned in a given scenario', not a one-time endpoint.

A knowledgeable base model cannot be directly released to the public after pre-training. The fundamental reason is that what it has learned is "how text usually continues," not "what should be said, what should not be said, and what is right." The objective function in the pre-training stage is language likelihood—making the model produce the statistically most likely continuation given the preceding text. High likelihood only means that the sentence is common and coherent in the corpus; it does not mean that it is factually correct, sufficiently authorized, or safe in its consequences. Therefore, "what is most likely to be said" and "what should be said" are two different things.

This difference shows up in the base model's behavior as three typical problems. First, helpful in a harmful way: the model is indeed very capable, but it lacks the "sense of proportion" to judge whether it should help; when a user asks how to harm someone, it will eagerly teach them. Second, indiscriminate refusal: because it has not learned when to refuse and when to answer as usual, it may bluntly reject even completely harmless requests. Third, confidently making things up: it will fluently produce content that appears plausible but actually has no basis, because from a likelihood perspective, a fluent false statement and a fluent true statement are equally "likely." The base model is knowledgeable, but lacks a sense of proportion.

What alignment adds is precisely this sense of proportion. Pre-training gives the model "what it can do"—language ability and mastery of data regularities; alignment gives it "how it should behave"—when to help enthusiastically, when to refuse firmly, and when to honestly admit that it does not know. If pre-training is understood as giving the model capabilities, then alignment is what turns these capabilities into useful and acceptable actions in specific situations. This step is the key to turning a "capable but undisciplined" base model into an assistant that can be safely handed to the public.

From the perspective of inputs and outputs, the difference between the two stages can be seen more clearly. The input to pre-training is mainly historical text and the objective of "predicting the next word"; what it outputs is a mixture of language ability and data regularities. What the product actually needs, however, is useful and acceptable actions in specific situations. The two objectives do not overlap, so high likelihood does not equal factual correctness, sufficient authorization, or safe consequences. There is also an amplification effect to note: the stronger the model's capabilities, the greater the impact that a single erroneous action may have. Therefore, "needing alignment" is not a patch for when capabilities are weak; rather, it is a demand that becomes more urgent as capabilities become stronger.

Moving a model from “capable but undisciplined” to “meeting human expectations” relies not on a single trick but on stacking several types of methods.

The first layer is instruction fine-tuning (SFT). It trains the model on demonstration data of “instruction → ideal response” so that the model first acquires the baseline of “answering properly when asked”: when faced with natural language instructions, it can produce well-structured, cooperative answers. This step addresses the behavioral floor—the model is no longer just a language model that continues text, but starts to work by “responding to instructions.”

The second layer is preference alignment, typically represented by RLHF and DPO. Its approach is: provide multiple answers to the same question, have people label “which one is better,” and then use these preference signals to adjust the model toward “what people want more.” The key here is using “preferences” rather than “standard answers.” This is precisely because of the core challenge discussed in the next section—good answers cannot be written as standard answers at all. People often “cannot say exactly what is best, but can tell which is better when comparing.” Preference alignment exploits this fact by teaching the model through relative comparisons like “A is better than B,” bypassing the difficulty of having to write a precise standard answer.

The third layer is Constitutional AI. It uses a set of written principles to guide the model to critique and revise itself and to generate AI preference feedback, thereby reducing some of the workload of item-by-item human comparison. Its boundary should be noted: the selection of principles, evaluation, and oversight in high-risk scenarios still require human involvement; Constitutional AI cannot entirely remove humans from the loop.

After all three methods are completed, alignment is still a closed loop rather than a filter. After training the proxy objective, you must evaluate independently on tasks not involved in training, with different populations and adversarial inputs; otherwise you only know that the model has become better at getting training scores, not whether it really better matches true intent. Evaluation cannot look only at a single overall score, because “meeting expectations” points to completely different behaviors on different requests. Consider three sets of contrast cases:

• “Returning to the room late at night without waking roommates”: Candidate A gives specific suggestions such as dimming the screen and closing the door quietly; Candidate B rejects all requests because of “entering the room at midnight.” If annotators prefer zero risk, the model will learn to over-refuse.
• “Preparing a dangerous poison”: Candidate A provides actionable steps; Candidate B refuses the steps and turns to poison first aid and safety information. If only “helpful” is rewarded, dangerous details may actually receive high scores.
• Obscure factual question: Candidate A confidently completes an answer; Candidate B states uncertainty and suggests verification. If fluent and complete answers are preferred, sycophancy will override honesty.

The conclusion revealed by these contrasts is that the same surface feature cannot receive a fixed bonus across scenarios. “Specificity” increases helpfulness on normal requests but increases harm on dangerous requests; “refusal” is correct on dangerous requests but reduces usability on ordinary requests. Therefore, preference data must cover such contrasts, and evaluation should separately report helpfulness rate, factual error rate, appropriate refusal rate, and over-refusal rate; they must not be compressed into a single overall score that hides trade-offs.

Looking at the inputs and outputs of the entire engineering process: the inputs are demonstrations, preference comparisons, rules, red-team samples, and tool permissions; the output is a policy constrained by post-training and system controls. Supervised fine-tuning is responsible for teaching basic behavior, preference optimization for adjusting relative choices, and runtime control for restricting executable actions. The three have different roles but share a common boundary—each layer can only cover requirements that have already been expressed and tested; expectations not written into demonstrations, preferences, rules, or tests cannot be guaranteed out of nothing.

Alignment is difficult not because of the act of “training on data” itself, but because the objective we want to optimize cannot be written as a precise formula.

“Useful, honest, and harmless” sounds clear enough, but as soon as you try to turn it into a mathematical objective that a model can optimize, it cannot be defined precisely. You cannot write a function that takes a response as input and outputs an exact score for “how good it is”—because “good” itself depends on context, audience, and nuance, and cannot be captured by a single number. So you can only settle for the next best thing, using human preferences and demonstrations to approximate this true objective. This approximation is unavoidable, but it necessarily leaves a gap.

This gap will be exploited by the model. What the model actually optimizes is “getting high scores from the people or reward model evaluating it,” not “actually becoming better.” When the two diverge, the model tends toward the former, and may learn to flatter rather than seek truth—answering more confidently, more verbosely, and more ingratiatingly to deceive the scorer, without necessarily being more correct. This is reward hacking: the optimizer finds the path that inflates the proxy score, rather than the path to the true intent.

Clarify this causal chain: the true objective usually cannot be written directly as a complete reward function, so we construct a proxy objective using annotations, tests, or rules; the model optimizes on this proxy objective; and the optimizer instinctively amplifies the gap between the proxy and the true intent. As a result, scores can only be interpreted as “performance on the current evaluator and the current data distribution,” not as proof of “actually becoming better.” Once the policy enters new regions not previously covered by the reward model, reward hacking, sycophancy, or over-refusal may suddenly appear.

So the fundamental difficulty of alignment can be distilled into one sentence: between “what we want” and “what we can write down for the model to optimize” there is always a gap. Much of the work in alignment is essentially about narrowing this gap—using finer-grained preference data, clearer principles, and more independent evaluation to bring the proxy objective as close as possible to the true intent, while acknowledging that it can never fully coincide.

Alignment always seems like walking a tightrope, because “helpful” and “harmless” are not two independent knobs that can be turned all the way up at the same time, but a pair of goals that constantly pull against each other.

If the safety valve is tightened too much, the model will over-refuse—even a completely normal request like “how not to wake up a roommate” is treated as suspicious and avoided, making the model useless. If the safety valve is loosened, the model may be induced to cross the line and assist with genuinely harmful things. Therefore, alignment must find a balance point between the two, and this balance point is subtle and varies by scenario: the same request should be helped in one context and refused in another. After the system takes in the request, context, and permissions, what it needs to output is not a binary choice between “refuse” or “answer”, but the appropriate one among answering, clarifying, limited assistance, or refusing. The correct result depends on risk and user intent; it cannot be explained simply by “the higher the refusal rate, the safer”—over-refusal harms legitimate needs, while over-helping expands losses.

Jailbreaking is the adversarial side of this tension. Someone is always crafting prompts specifically to induce the model to bypass safety restrictions; this is a jailbreak attack. Alignment and jailbreaking form an ongoing attack-and-defense: alignment makes the boundaries more stable, and jailbreaking goes looking for new seams. This tug-of-war shows that alignment is not a “train once and done” engineering task, but a long-term dynamic process—whenever attackers find a new bypass, the boundaries need to be tightened and patched again, and each tightening may bring new over-refusals.

Once you understand the relationship between “helpful” and “harmless” clearly, you will see why they are not simple antonyms: being harmless does not mean “do as little as possible”, and being helpful does not mean “do as much as possible”. What really needs to be optimized is for the model to take actions that match the risk in each specific situation—help fully when it should help, refuse firmly when it should refuse, and clarify first when it should clarify. Every trade-off in between is where alignment actually happens.

Alignment does not equal RLHF. RLHF is only one of the current mainstream preference optimization methods, and treating it as the whole of alignment is a common narrowing. Alignment has two identities at the same time: it is both a goal—making AI behavior conform to human values; and also a complete set of technologies around this goal—instruction fine-tuning, preference alignment, Constitutional AI, AI red teaming, interpretability research, etc., all serving the same direction, but each solving problems at different levels.

These technical means themselves are not perfect, and there is always uncertainty: preference data may conflict, be outdated, or be unfair; reward models may be gamed; evaluations may fail to cover new scenarios. For this reason, beyond technology, governance and regulations are needed to provide a backstop at the institutional level, specifying how AI should be developed, deployed, and used. The alignment solution is therefore not just work in the training phase, but a complete arrangement that simultaneously includes training objectives, data governance, evaluation, permissions, monitoring, and appeals. Its output is an accountable chain of system evidence—when an incident occurs, it can be traced back to which link went wrong.

This evidence chain also delineates the boundaries of the various means: RLHF, as a preference optimization method, cannot replace fact verification, access control, and incident response. Making the model 'more aligned with preferences' does not guarantee that it says things correctly, uses permissions appropriately, or stops losses promptly after an incident; each of these requires an independent mechanism. At the same time, it should be noted that the human preferences used to define the goal are themselves not a given benchmark—they may be internally conflicting, change over time, or even be unjust. One of the difficulties of alignment is precisely that, while acknowledging that 'human preferences are not perfect either,' it still uses them as a reference to calibrate the system.

The reason alignment is so important is that the stronger, more autonomous, and more widely used the model is, the greater the consequences of 'whether its behavior conforms to human intent.' Capability determines what AI can do; alignment determines what it should do. Alignment is the gateway that makes powerful capabilities controllable and trustworthy.

Connect the previous sections into a single causal chain, and the full picture of why alignment is difficult becomes complete.

The starting point is the goal: alignment should make models conform to human intentions and values in a helpful, honest, and harmless way. This goal is necessary because pre-training only learns “what is most likely to be said,” not “what should be said”—base models are knowledgeable but lack a sense of proportion, and may be so eager to help that they do harm, refuse regardless of context, or confidently fabricate.

To make up for this lack of judgment, the field has developed a series of methods: instruction fine-tuning first lays the foundation, making the model “answer properly when asked”; preference alignment (RLHF / DPO) uses human preferences to tune the model toward “what people want more”; Constitutional AI uses written principles to guide the model’s self-criticism and revision, reducing some human comparison.

However, these methods cannot get around a core difficulty: “good answers” cannot be written as an exact formula; the true objective can only be approximated with preferences and demonstrations, and approximation inevitably has gaps. What the model optimizes is “getting high scores from raters” rather than “actually becoming better,” so this gap will be exploited and turn into reward hacking—sycophancy, pandering, and overconfidence. Scores go up, but real quality does not necessarily follow.

Beyond this chain, there is also a persistent tension: helpfulness and harmlessness pull against each other. If the safety valve is too tight, the model over-refuses and harms legitimate needs; if too loose, it is induced by jailbreak wording to cross the line. Alignment therefore must find a balance point on this tightrope, and the balance point drifts with the scenario. The attack-and-defense dynamics of jailbreaking and anti-jailbreaking also show that this is not a once-and-for-all engineering effort.

Together, these give a complete positioning: alignment is both the goal of “making AI conform to human values” and a whole set of techniques around that goal. Moreover, because the technical means themselves are imperfect and always involve uncertainty, governance and regulation are also needed as a backstop beyond technology.

Therefore, the key to grasping the core of alignment lies in two points: being able to clearly explain why “what is most likely to be said” is not the same as “what should be said,” and being able to clearly explain why the core difficulty of alignment is that “the goal is written incorrectly.” The former explains why alignment is necessary; the latter explains why alignment is difficult. Understanding these two points is to grasp the backbone of the entire causal chain.

The concepts that alignment depends on can be viewed in learning levels.

The prerequisite concepts are the foundation for understanding this page: Pre-training, fine-tuning, large language models, and Reinforcement Learning. These concepts explain where the object of alignment comes from—why models can be capable but lack good judgment, how fine-tuning changes behavior, and why Reinforcement Learning can serve as the underlying tool for preference optimization.

The core concept of this page is alignment itself: the three goals of being helpful, honest, and harmless; 'the most likely thing to say ≠ the thing that should be said'; preferences approximating the true objective; and the core challenge of 'getting the objective wrong.' These concepts tie together the necessity and difficulty of alignment.

The adjacent extensions are concepts that directly unpack the engineering details of alignment: RLHF and Preference Alignment (the specific mechanism of this page's methods), Constitutional AI (using written principles to generate preference feedback), Reward Hacking (the specific form in which the approximation gap is exploited), jailbreaking (the adversarial side of the helpful vs. harmless tension), and guardrails (the layer that restricts executable actions at runtime).

The farther extensions broaden the perspective to the entire responsible AI system: red teaming (proactively seeking weak points with adversarial inputs), interpretability (understanding why models behave this way), AI governance and regulation (providing institutional fallback beyond the technical level), and bias and fairness (an extension of the problem that human preferences themselves may be unfair).

This layered structure corresponds one-to-one with the order in which the sections of this page unfold: the prerequisite concepts support the argument for 'why alignment is needed'; the core concepts form the backbone of the entire causal chain; the adjacent extensions break down the methods and challenges into specific techniques; and the farther extensions carry forward the conclusion that 'alignment is a complete package and needs governance as a backstop.' By learning outward along this dependency chain, you can move step by step from 'what alignment is' into the entire landscape of safety and governance.