Automated Check
A machine verdict the agent cannot talk its way past.
Tests, linters, type checks, schema validation, build steps — deterministic signals that say whether the work is acceptable. The backbone of trusting unattended work.
The stronger your checks, the more autonomy you can safely grant. This is the real lever on agent autonomy: not model quality, but whether failure is cheap and detectable.
Deterministic checks are what make autonomy defensible, because they are the part of the system that cannot be talked into a different opinion. The strength of your unattended workflow is essentially the strength of its checks — everything else is a claim.
Turning a requirement into a gate
The agent must not change the public API.
A type check or an API-diff test enforces it objectively. A sentence in the prompt asks for it. Where the requirement is load-bearing, put it in a check, because that is the version that cannot be forgotten under context pressure.
- blogDemystifying evals for AI agents ↗
Gives working definitions for harness, task, trial, grader, transcript and outcome, and argues you must evaluate the model and harness together and judge final environment state, not just the last message.
- blogBuilding effective agents ↗
Draws the line between fixed workflows and agents that choose their own process, and names the core patterns: chaining, routing, parallelisation, orchestrator-workers, evaluator-optimizer.
Automated Review
A model reviewing a model’s output before a human sees it.
AFK · Away from keyboard
Running without a person watching.
Regression Test
A test that catches a specific past failure coming back.
Reward Hacking · Goodharting
Optimising the metric instead of the goal.