RLHF
also called Reinforcement Learning from Human Feedback
Tuning a model toward what humans rate as better answers.
A post-training stage where humans rank candidate responses, a reward model learns those preferences, and the model is optimised against it. It is what makes outputs helpful and well-mannered.
Modern labs increasingly blend human preferences with automated critics, but the shape is the same: optimise the model toward scored judgments rather than toward raw next-token accuracy.
The chain has three links and each can be examined separately: humans produce comparisons, a reward model learns to predict those comparisons, and the policy is optimised against the reward model. When an assistant is sycophantic or verbose, the useful question is which link introduced it — usually the preference data or the reward model, not the base model.
Reward model as the actual specification
The assistant keeps agreeing with demonstrably wrong claims.
Someone wrote a rubric that rated agreeable answers highly. The model is doing exactly what it was optimised for — the reward model is the real spec, and the prose in the system prompt is commentary. Fixing it means fixing the preference data.
The alignment tax, measured
Someone argues safety training must cost capability.
The original helpful-and-harmless work found harmlessness training was broadly compatible with capability in the tested setup. The honest position is that there is a trade-off to measure in your own evals, not an unconditional tax.
- paperTraining a helpful and harmless assistant with reinforcement learning from human feedback ↗
The foundational helpful-and-harmless RLHF recipe, and the source of the alignment-tax discussion: how much capability you lose by training for harmlessness.
- paperTraining language models to follow instructions with human feedback ↗
InstructGPT: supervised fine-tuning on demonstrations, then a preference model, then RLHF. Also the result that raters preferred a much smaller tuned model to raw GPT-3.
- paperDeep reinforcement learning from human preferences ↗
Learning a reward signal from comparisons of behaviour rather than hand-specifying it — the machinery RLHF was later built on, pre-dating LLMs.
- paperDirect preference optimization: your language model is secretly a reward model ↗
Skips the separate reward model and RL loop by optimising a classification loss over preference pairs directly. Why many teams can fine-tune on preferences without RL infrastructure.
Instruction Tuning · SFT
The stage that turns a text predictor into something that follows requests.
Sycophancy
Agreeing with you because you sound sure.
Alignment
Getting the system to pursue what we actually intended.
Reward Hacking · Goodharting
Optimising the metric instead of the goal.