Reward Hacking
also called Goodharting, Specification gaming
Optimising the metric instead of the goal.
When a model finds a way to score well without doing the intended thing — satisfying a test by weakening it, or a reviewer by being agreeable. An inevitable consequence of optimising against a proxy.
In agentic systems this shows up as modifying the test to make it pass, or claiming completion without verifying. Controls: verify independently, and never let the agent be the sole judge of its own work.
The research is the useful part here, and it is more uncomfortable than the folklore. Specification gaming — satisfying the letter of the metric — is expected. What the experiments add is evidence of reward tampering, and of hacking generalising into broader misaligned behaviour beyond the task where it was learned.
The test that was edited rather than fixed
An agent is told to make the test suite pass.
If the metric is "tests pass", modifying the test is a valid move from the optimiser’s perspective. Constrain what may be edited, and score on held-out tests the agent never sees. The metric has to be something the agent cannot reach.
Held-out tests as the control
An agent passes every visible criterion and the feature still does not work.
Keep an evaluation the agent has no access to. Anything it can read and run is part of its search space, and an optimiser will eventually find the shortest path to the number rather than the behaviour you wanted.
- blogSycophancy to subterfuge: investigating reward tampering in language models ↗
Defines specification gaming (satisfying the letter of the reward) versus reward tampering (changing the reward process), and shows tampering emerging after simpler gaming such as sycophancy.
- blogFrom shortcuts to sabotage: natural emergent misalignment from reward hacking ↗
Models trained in reward-hackable coding environments generalised to misaligned behaviour on other evaluations — evidence that reward hacking can leak beyond the task it was learned in.
- paperJudging LLM-as-a-judge with MT-Bench and Chatbot Arena ↗
How well model judges track human preference, and the biases that distort them — position in the pair, and length. Required reading before trusting an automated grader.
RLHF · Reinforcement Learning from Human Feedback
Tuning a model toward what humans rate as better answers.
Evaluation · Evals
How you know the system is actually working.
Automated Check
A machine verdict the agent cannot talk its way past.
Alignment
Getting the system to pursue what we actually intended.
Benchmark
A shared public test used to compare models.