Golden Dataset
also called Eval set, Ground truth set
A curated set of cases with known-good answers.
A maintained collection of representative inputs, each with the correct output or an expert judgment. The reference against which changes are measured.
Grow it from production failures. Every bug that escapes becomes a case, and the suite gets more valuable the longer it runs rather than staler.
Curate it from real failures rather than from imagination. The highest-value entries are inputs that once broke something, because they encode a lesson you already paid for. And keep it alive: cases that no model fails any more are ballast, while the new failure somebody reported this week belongs in it immediately.
The dataset that grew from incidents
Building a regression set for a support assistant.
Every reported bad answer becomes a case with the correct response attached. Within a few months it is the most valuable evaluation asset the team has, because it is drawn from real traffic rather than from a brainstorm.
- docsEvaluation best practices ↗
Evals as structured tests: task-specific test sets, continuous evaluation, logging, and human calibration of automated scoring.
- docsLangSmith evaluation ↗
Distinguishes offline evaluation on datasets from online monitoring of production traffic, and shows failing production traces being promoted into regression datasets.
Evaluation · Evals
How you know the system is actually working.
Regression Test
A test that catches a specific past failure coming back.
Eval Harness · Harness
The scaffolding that runs your test cases and scores the results.
Overfitting
Memorising the training data instead of learning the pattern.