Constitutional AI
Training a model to critique itself against written principles.
An approach where a model is given an explicit set of principles and trained to revise its own responses to comply with them, reducing reliance on human labelling for every judgement.
The lasting idea is that written principles are auditable artifacts. You can read them, argue about them, and change them — unlike a preference baked silently into a reward model.
The contribution worth understanding is the substitution: written principles plus model self-critique in place of human labels identifying harmful output. That makes the value system explicit, reviewable and changeable, which is a large practical advantage over an unlabelled preference dataset.
Making the value system readable
A team is asked what their assistant will and will not do.
With principles in the training loop, the answer is a document that can be read and argued with. That is a governance improvement independent of any capability claim, and it is why the approach is widely discussed even outside the labs that use it.
- paperConstitutional AI: harmlessness from AI feedback ↗
Replaces human harm labels with a written set of principles, a self-critique and revision pass, then reinforcement learning from AI feedback.
- paperTraining a helpful and harmless assistant with reinforcement learning from human feedback ↗
The foundational helpful-and-harmless RLHF recipe, and the source of the alignment-tax discussion: how much capability you lose by training for harmlessness.
Alignment
Getting the system to pursue what we actually intended.
RLHF · Reinforcement Learning from Human Feedback
Tuning a model toward what humans rate as better answers.
Guardrails
Constraints on what the system will do, enforced in code.
Content Policy
The written rules about what the system may produce.