Regression Test
A test that catches a specific past failure coming back.
A test case derived from a bug or bad output that asserts it no longer occurs. For AI systems, it asserts on behaviour or properties rather than exact strings.
Because output is non-deterministic, AI regression tests assert on structure, constraints and judged quality — never on a byte-for-byte match.
For generative systems the assertion has to move from value to property. You cannot assert an exact string, but you can assert that it parses, contains a required field, cites a source, refuses what should be refused, and does not leak a specific forbidden string.
Properties instead of golden strings
A summariser’s output changes every run.
Assert the properties: under the length cap, contains the three required entities, makes no claim absent from the source, and names the decision. Stable enough to run in CI, and still catches the regression that matters when someone edits the prompt.
- docsEvaluation best practices ↗
Evals as structured tests: task-specific test sets, continuous evaluation, logging, and human calibration of automated scoring.
- docsLangSmith evaluation ↗
Distinguishes offline evaluation on datasets from online monitoring of production traffic, and shows failing production traces being promoted into regression datasets.
- docsUsing GitHub Copilot code review ↗
Details the boundary their own review agent draws by default: comments rather than approvals, with approval rights switched on deliberately.
Golden Dataset · Eval set
A curated set of cases with known-good answers.
Evaluation · Evals
How you know the system is actually working.
Non-determinism
The same input can produce different output, and that is by design.
Drift
Quiet degradation as the world, or the model, changes under you.