Alignment
Getting the system to pursue what we actually intended.
The problem of ensuring a model’s goals and behaviour match human intent — including in situations the designers did not anticipate. Broad enough to cover everything from tone to catastrophic risk.
For product teams the practical face of alignment is narrow: does the system do the requested thing, refuse the unrequested thing, and behave predictably when uncertain?
The useful split for anyone shipping a product is between the problems you can address at your layer and the ones you cannot. Your layer holds the harness: permissions, sandboxing, approvals, monitoring, and the decision about how much autonomy to grant. The training-side problems are inherited, which is why containment matters even when you trust the model.
Measuring the trade rather than debating it
A model refuses too many legitimate requests after a safety update.
The safety-usability trade-off is measurable: over-refusal rate on your real traffic against the failure rate you were protecting against. The classifier-based defences were published with exactly that trade-off documented, which is the honest way to discuss it.
- paperTraining a helpful and harmless assistant with reinforcement learning from human feedback ↗
The foundational helpful-and-harmless RLHF recipe, and the source of the alignment-tax discussion: how much capability you lose by training for harmlessness.
- paperConstitutional AI: harmlessness from AI feedback ↗
Replaces human harm labels with a written set of principles, a self-critique and revision pass, then reinforcement learning from AI feedback.
- reportAnnouncing our updated Responsible Scaling Policy ↗
Capability thresholds with graduated safeguards and documentation duties, written in the safety-case style: here is the evidence that this risk is adequately handled.
RLHF · Reinforcement Learning from Human Feedback
Tuning a model toward what humans rate as better answers.
Guardrails
Constraints on what the system will do, enforced in code.
Reward Hacking · Goodharting
Optimising the metric instead of the goal.
Refusal
The model declining a request.
Constitutional AI
Training a model to critique itself against written principles.