Human Eval
also called Human review, Manual grading
People scoring the output — the ground truth everything else is calibrated against.
Evaluation performed by people, usually on a sample, with a rubric. Slow and expensive, but it is the only way to know whether automated scores mean anything.
Use human review to calibrate the judge, not to score everything. Twenty well-designed human labels can validate a scoring rubric that then runs on twenty thousand cases.
The reason to spend on it is not the score, it is the calibration. A few hundred carefully labelled examples tell you whether your automated metric tracks anything real, which is what licenses you to trust that metric on the thousands of cases you cannot afford to read.
Spot-checking the grader
An automated quality metric has been improving for weeks.
Read fifty outputs from the current system and fifty from three weeks ago, blind. If the improvement is not visible to a person, you have been optimising the metric. Finding that out early is what prevents a confident release of something that never got better.
- paperJudging LLM-as-a-judge with MT-Bench and Chatbot Arena ↗
How well model judges track human preference, and the biases that distort them — position in the pair, and length. Required reading before trusting an automated grader.
- blogIntroducing Bloom: an open source tool for automated behavioural evaluations ↗
An agentic framework that generates behavioural eval scenarios and scores them with model judges, with a human-calibration check and a warning that eval designs go stale.
- blogA statistical approach to model evaluations ↗
Why a benchmark-score difference may be noise: report uncertainty, account for clustered questions, use paired comparisons.
LLM as Judge · Model grading
Using a model to score another model’s output.
Evaluation · Evals
How you know the system is actually working.
Human Review
A person reading the work, with judgment as part of the job.
Benchmark
A shared public test used to compare models.