Eval Harness
also called Harness
The scaffolding that runs your test cases and scores the results.
The code that feeds cases to the system, collects outputs and applies scoring — exact match, rubric, model judge or human. Usually a small purpose-built script plus a place to store results.
The valuable property is repeatability: the same suite, run the same way, compared over time. A spreadsheet of manual checks decays within a month.
Keep it boring and keep it runnable by anyone. The value is not sophistication, it is that a change can be scored in minutes by a person who did not write it. A harness that needs the author to run it is a harness that will be used once.
Owning the whole environment
Evaluating a coding agent on multi-step tasks.
The grader has to inspect the resulting repository, not the final message: did the tests pass, is the file changed, did it avoid the protected directory. That is a different kind of assertion from single-turn output scoring, and it is where harness quality shows.
- blogDemystifying evals for AI agents ↗
Gives working definitions for harness, task, trial, grader, transcript and outcome, and argues you must evaluate the model and harness together and judge final environment state, not just the last message.
- docsLangSmith evaluation ↗
Distinguishes offline evaluation on datasets from online monitoring of production traffic, and shows failing production traces being promoted into regression datasets.
Evaluation · Evals
How you know the system is actually working.
Golden Dataset · Eval set
A curated set of cases with known-good answers.
LLM as Judge · Model grading
Using a model to score another model’s output.
Regression Test
A test that catches a specific past failure coming back.