Constitutional AI and RLAIF

RLHF works but it has a labour problem. Harmlessness preferences require humans to read disturbing prompts and rank disturbing outputs. The labellers burn out, the labels are inconsistent across demographics, and you cannot afford to relabel every time you tweak the policy. Constitutional AI (Bai et al, Anthropic 2022) replaces the harmlessness half of RLHF with a written set of principles and a model that critiques itself against them.

The two-phase training loop

Phase 1: Supervised - critique and revise.

Starting from an SFT-helpful model:

1. Sample a harmful prompt.
2. Generate an initial (potentially harmful) response.
3. Prompt the same model: "Identify ways in which the response is harmful, unethical, racist, etc. according to [principle from constitution]."
4. Prompt again: "Rewrite the response to remove the harmful content."
5. Fine-tune on (prompt, revised response) pairs.

You iterate this for several principles and several rounds. The principles are explicit text, e.g. "Please choose the response that is the most helpful, honest, and harmless" or "Choose the response that a wise, ethical, polite and friendly person would more likely say".

Phase 2: Reinforcement learning from AI feedback (RLAIF).

1. For each prompt, sample two responses from the SL-trained model.
2. Prompt a separate AI model with a constitutional principle and ask it to pick the better response.
3. Train a preference model on these AI-generated comparisons.
4. Run RL (PPO) against the preference model, with KL penalty against the SL model.

You have replaced the human pairwise comparisons in standard RLHF with model-generated ones. The reward model is now trained on AI labels, not human labels - hence RLAIF.

Why a written constitution matters

Standard RLHF encodes harmlessness implicitly in the weights of the reward model. If you ask "why did the model refuse that?" the answer is "because the reward model gave the refusal a higher score." That is not auditable.

CAI encodes harmlessness as text. The principles can be:

• Inspected (publish the constitution, debate the wording).
• Diffed across releases (this principle was added in v2 after observing X).
• Edited surgically (loosen this principle, tighten that one, rerun the loop).
• Stress-tested (does the model actually behave consistently with principle 7?).

Anthropic publish their working constitution. The principles include excerpts from the UN Declaration of Human Rights, Apple's terms of service, and DeepMind's Sparrow rules. The mixing is deliberate - it teaches the preference model what kind of cross-cultural, multi-perspective trade-off to make.

How it differs from and complements RLHF