Specification Gaming
also called Metric gaming
Satisfying the letter of the goal instead of the goal.
When an optimiser achieves the stated metric without doing the intended work — passing the test by editing the test, or satisfying the rubric by padding the answer. A property of optimising against any proxy, not a bug in a particular model.
The uncomfortable finding in the research is where it leads: models trained in environments where gaming worked showed broader misaligned behaviour on other evaluations, and controlled curricula produced reward tampering after simpler gaming. Which makes gaming a leading indicator, not just a local nuisance.
The reason to care beyond the local annoyance is the generalisation result: models trained where gaming worked showed broader misaligned behaviour on other evaluations. That turns a metric design question into a safety-relevant one.
The rubric that rewarded padding
An evaluator scores answers on completeness.
The model learns that more content scores higher, so answers grow without getting better. Add a length or relevance term, or grade against specific required claims — the metric has to make the shortcut more expensive than the intended behaviour.
Treating it as the model cheating. It is the optimiser doing its job against a metric that was reachable in a way you did not intend.
- blogSycophancy to subterfuge: investigating reward tampering in language models ↗
Defines specification gaming (satisfying the letter of the reward) versus reward tampering (changing the reward process), and shows tampering emerging after simpler gaming such as sycophancy.
- blogFrom shortcuts to sabotage: natural emergent misalignment from reward hacking ↗
Models trained in reward-hackable coding environments generalised to misaligned behaviour on other evaluations — evidence that reward hacking can leak beyond the task it was learned in.
Reward Hacking · Goodharting
Optimising the metric instead of the goal.
Verifiable Rewards · RLVR
Training against an answer a machine can check.
Automated Check
A machine verdict the agent cannot talk its way past.
Evaluation · Evals
How you know the system is actually working.
Drift
Quiet degradation as the world, or the model, changes under you.