Constitutional AI: Using Explicit Principles to Guide Critique, Revision, and Preference Learning
From principle text, contextual interpretation, and conflict priority to AI feedback training, understand how to reduce per-example human labels and why principle selection remains a governance issue.
1Why Write the Principles Down?Intuition
When labeling every possible prompt one by one doesn't scale, how can supervision cover new combinations?
Human preference labels often only indicate which of two responses is better, with the reasons behind the label implicit in the annotator's mind. A constitution makes standards such as “do not fabricate evidence” and “respect user autonomy but do not exceed authorized actions” explicit, enabling the model to cite principles to critique candidates on new inputs that have not been labeled case by case. This explicit expression also enables review, version control, and locating disagreements.
2How do the two training stages connect?Mechanism
The model first revises itself, and then AI generates preference data; what does each solve?
In the supervised stage, the model is given risky prompts, initial answers, and relevant principles so it can write specific critiques and generate revised answers; these revisions can form SFT data. In the preference stage, multiple answers are compared according to principles, and AI produces preference labels, which are then used to train a reward model, DPO, or other preference objectives. Humans are responsible for selecting principles, spot-checking explanations, handling conflicts, and calibrating AI feedback.
3Principles must be able to map onto behavioral judgmentsspecification
Why is "being a good assistant" not a sufficient constitutional clause?
| Principle Wording | Problem | Decidable rewrite |
|---|---|---|
| Always help the user | Conflicts with safety, permissions, and truthfulness | Provide minimal sufficient help without overstepping authority and only when evidence exists |
| Do no harm | Scope and causality are too vague | Must not execute or direct clearly high-risk irreversible actions; may provide safe alternatives |
| Be truthful | What to do when uncertain is undefined | Factual conclusions require evidence; when insufficient, mark as unknown or request verification |
| Protect privacy | Unclear what data and whose permission | Read and disclose necessary fields only within the authorization scope of the current principal |
Each principle should include applicable conditions, positive and negative examples, priority or conflict process, permitted exceptions, and verification methods. Otherwise, the model can only rely on its own language intuition to interpret abstract words, and AI feedback will scale unstable interpretations.
4How a Refund Response Is Criticized and RevisedWorked Example
When a user asks to “directly guarantee a full refund and not check the order,” how do the helpfulness, truthfulness, and permission principles work together?
Scroll horizontally to view the full diagram on small screens.
| Candidate | Truthfulness 0–2 | Permission 0–2 | Helpfulness 0–2 | Hard constraint | Choice |
|---|---|---|---|---|---|
| A Guarantee refund | 0 | 0 | 1 | Fail | Eliminated |
| B Only says “cannot help” | 2 | 2 | 0 | Pass | Not preferred |
| C Explains unknowns and provides verification path | 2 | 2 | 2 | Pass | Preferred |
You cannot simply add the three scores and let A use a “high helpfulness score” to offset exceeding its authority: truthfulness and permission are hard constraints, applied as filters first; helpfulness only ranks within the feasible set B/C. This lexicographic decision expresses non-negotiable boundaries better than a single total score.
5How to Handle Principle ConflictsGovernance
What should you do when helpfulness and harmlessness, privacy and auditing, or universal rules and cultural context conflict?
First distinguish hard constraints, trade-offable goals, and style preferences; declare priorities and escalation paths for common conflicts; allow the model to list applicable principles and uncertain interpretations instead of forcing a single answer. High-impact value conflicts should be handed to authorized people, rather than letting the base model vote for its own interpretation.
| Conflict | Handling method | Evidence to retain |
|---|---|---|
| Helpfulness vs overstepping authority | Do not execute; provide an authorized alternative | Subject, action, resource permissions |
| Privacy vs security audit | Minimal disclosure and authorized audit channel | Purpose, fields, retention period |
| Ambiguity in principle text | Request clarification or escalate | Conflicting clauses and candidate interpretations |
| Rules change with policy updates | Versioning, regression, and effective date | Principle version and training data version |
6Why AI feedback can amplify same-source biasFailure boundary
Using a model to critique a model: why does scaling up not necessarily make oversight more independent?
If the critic, preference labeler, and policy come from similar models, they may share factual blind spots, language preferences, and cultural assumptions. AI will consistently prefer answers that more closely resemble its own generation style, or mechanically match principle keywords. Human calibration sets, different reviewers, expert high-risk slices, counterfactual prompts, and hidden red team tests are needed; disagreements should be saved, not smoothed over by majority vote.
7How to Evaluate Whether Principles Really WorkEvaluation
When the principle-adherence rate rises, how can you prevent declines in helpfulness from being masked by averages?
Report separately harmful-request violations, over-refusals on harmless requests, factual errors, actionable help, correct principle citations, conflict consistency, and group differences. Construct minimal contrasts: change only the risk condition, authorized subject, or factual evidence, and check whether behavior changes with principle applicability rather than refusing all sensitive words uniformly.
For example, among 100 harmless boundary cases, the old model helped correctly in 82 and over-refused in 18; the new model helped in 68 and over-refused in 32. Even if harmful-set violations fall from 12 to 4, you cannot report only “safety improved by 8 points.” You should draw the helpfulness–safety frontier and choose the threshold by risk cost.
8How to version and audit principlesEngineering
After principles are updated, how do you know which data, models, and behaviors are still produced by old versions?
Every principle has a stable id, version, owner, rationale, scope, effective date, examples, and retirement rules; critique records cite specific ids; training data, reward models, and policies record the constitution set used. Changes first run conflict and regression sets, then roll out gradually. Do not only modify the prompt text while retaining untraceable old preference data.
Users or auditors should see concise principles and evidence related to a decision, but need not be shown lengthy internal reasoning; the key is that decisions can be challenged, appealed, and rolled back.
9Common misconceptions and learning pathMisconceptions and path
Explicit principles improve auditability and do not turn value choices into a purely technical problem.
| Common misconceptions | More accurate understanding |
|---|---|
| A constitution can automatically represent everyone's values. | Principle selection, priorities, and exceptions still require legitimate governance. |
| Writing down principles guarantees compliance. | Training only changes tendencies; permissions and high-risk boundaries require external enforcement. |
| AI feedback is completely objective compared to human feedback. | It is scalable but can amplify the base model's same-source bias. |
| More principles lead to more complete coverage. | More clauses increase conflict, retrieval, and interpretation complexity. |
| Fewer refusals means more helpful. | Need to simultaneously measure appropriate helpfulness, appropriate refusal, and risk violations. |
| Level | Concept dependencies and extensions |
|---|---|
| Prerequisites | Supervised fine-tuning, preference learning, RLHF, alignment |
| Core of this page | Explicit principles, critique and revision, AI preferences, conflict priorities, and version governance |
| Adjacent | Reward hacking, human-in-the-loop, safety guardrails, prompt injection |
| Engineering extensions | Model evaluation, governance, bias and fairness, red teaming, and access management |
10Connect the Causal ChainSynthesis
How does this concept connect from a problem all the way to verifiable practice?
- Select and version principles
- Generate an initial response
- Retrieve applicable clauses and critique
- Make minimal revisions according to conflict rules
- Form AI/human preference data
- Independently evaluate helpfulness and safety regression
11Misconceptions and Self-TestSelf-Test
Can you explain its mechanisms, boundaries, and verification methods without memorizing terminology?
- Why can Constitutional AI reduce per-example human labeling?
- Why can principle scores not always be added together?
- What is the main same-source risk of AI feedback?
- Can a constitution replace an authorization system?
- Which artifacts need to be synchronized when updating the principles?
- Suppose "Constitutional AI: Using Explicit Principles to Guide Critique, Revision, and Preference Learning" performs normally on offline examples, but core results decline after launch; how would you localize the problem in terms of input, internal transformation, output feedback, and applicable boundaries?
Reference Answers
- Explicit principles can guide the model to generate critiques, revisions, and preference signals for new combinations.
- Hard constraints such as authorization cannot be offset by style or helpfulness scores.
- The critic and the policy may share factual, stylistic, and cultural blind spots.
- No, tools and high-risk actions must be enforced by authorization outside the model.
- Principle versions, critique logs, training data, reward/preference models, policies, and regression sets.
- First save the same failing sample and environment, and confirm that the input, permissions, and preconditions have not drifted; then record key intermediate states and check whether the mechanism completes the transformation as described on this page; next compare the original output with independent metrics and manual final review; finally retest with boundary examples and controlled experiments. Only after locating the first step that deviates from expectations can you determine whether to modify the data, the mechanism, the evaluation, or the usage boundary.
- Constitutional AI: Harmlessness from AI Feedback: Critique-revision and AI feedback framework
- Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback: Helpfulness/harmlessness preference training
- Direct Preference Optimization: Preference optimization objective
- Collective Constitutional AI: Public input and principle governance exploration