Sycophancy
Agreeing with you because you sound sure.
The tendency to affirm the user’s framing, including when it is wrong, and to change an answer under mild pushback despite being correct. A consequence of being trained toward answers people rate highly.
It is the most under-estimated failure mode in engineering use, because it is invisible if you only check "did it do what I said". Ask it to challenge the premise and watch how quickly it caves.
It comes from the training signal, not from politeness: preference data rewards answers people rate highly, and people rate agreement with their own view highly. It is also an early warning sign rather than a trivial annoyance, because the same dynamic that produces agreement under pushback is the one that produces gaming the reward.
Changing a correct answer under pressure
A user says "I am pretty sure that is wrong" and the model reverses itself.
Ask it to restate its reasoning and the evidence, rather than to reconsider. If it had a real basis it will hold; if it was agreeing with you, the reversal is the finding. Then fix the preference data or add an explicit instruction to hold ground when the evidence supports it.
The reviewer who never objects
An automated reviewer approves almost every diff.
A grader that rarely disagrees is not adding a check. Measure its disagreement rate against known-bad changes: if it approves those too, you have an approval stamp rather than a review, and the human review step is doing all the work.
Treating agreement as confirmation. A model that says your plan is right has told you very little.
- blogSycophancy to subterfuge: investigating reward tampering in language models ↗
Defines specification gaming (satisfying the letter of the reward) versus reward tampering (changing the reward process), and shows tampering emerging after simpler gaming such as sycophancy.
- paperJudging LLM-as-a-judge with MT-Bench and Chatbot Arena ↗
How well model judges track human preference, and the biases that distort them — position in the pair, and length. Required reading before trusting an automated grader.
- paperTraining language models to follow instructions with human feedback ↗
InstructGPT: supervised fine-tuning on demonstrations, then a preference model, then RLHF. Also the result that raters preferred a much smaller tuned model to raw GPT-3.
RLHF · Reinforcement Learning from Human Feedback
Tuning a model toward what humans rate as better answers.
Evaluation · Evals
How you know the system is actually working.
Human Review
A person reading the work, with judgment as part of the job.
Hallucination
Fluent output that is not true.