LLM as Judge
also called Model grading
Using a model to score another model’s output.
Automated evaluation where a model compares or rates responses against a rubric. Cheap and scalable, and the only practical way to score free-form output at volume.
Known biases: preference for longer answers, for its own style, and for the first option presented. Mitigate with explicit rubrics, randomised ordering, and periodic calibration against human labels.
It is the only practical way to score free-form output at volume, and it carries known biases: it favours longer answers, and it is sensitive to the order in which candidates are presented. Both are correctable, and both matter more at small differences than large ones.
The judge that liked long answers
A judge scores verbose responses higher than concise ones in a style comparison.
Test the judge itself: give it pairs that are equal in quality but different in length and see whether it has a preference. If it does, score both orders, randomise presentation, or grade against a rubric that names the criteria instead of asking which is better.
Calibrating against people
Nobody knows whether the automated scores mean anything.
Hand-label a sample of a few hundred outputs yourself and check agreement with the judge. Where it disagrees, refine the rubric rather than the sample. A judge nobody has calibrated is a number, not a measurement.
- paperJudging LLM-as-a-judge with MT-Bench and Chatbot Arena ↗
How well model judges track human preference, and the biases that distort them — position in the pair, and length. Required reading before trusting an automated grader.
- blogIntroducing Bloom: an open source tool for automated behavioural evaluations ↗
An agentic framework that generates behavioural eval scenarios and scores them with model judges, with a human-calibration check and a warning that eval designs go stale.
- docsEvaluation best practices ↗
Evals as structured tests: task-specific test sets, continuous evaluation, logging, and human calibration of automated scoring.
Evaluation · Evals
How you know the system is actually working.
Human Eval · Human review
People scoring the output — the ground truth everything else is calibrated against.
Eval Harness · Harness
The scaffolding that runs your test cases and scores the results.
Bias · Fairness
Systematic skew in outputs, usually inherited from data.