Non-determinism
The same input can produce different output, and that is by design.
The property that identical requests may yield different results, because generation samples from a probability distribution and, in production, requests are batched across different hardware states.
Even at temperature zero, floating-point arithmetic and batching mean bit-for-bit reproducibility is not guaranteed across runs. Treat outputs as stochastic and evaluate statistically.
Assume that identical requests will differ, and build accordingly: evaluate over multiple samples rather than one, log enough to reproduce the inputs, and never write a test that asserts an exact string from a generative call. Assert on properties instead.
The test that fails one run in ten
A regression test compares the model output to a stored golden string.
It will flake, and the team will start ignoring it. Assert on what actually matters: the response parses against the schema, contains the required fields, cites a source, and refuses the input that should be refused. That test is stable and still catches real regressions.
Writing tests that assert on exact model output.
- docsEvaluation best practices ↗
Evals as structured tests: task-specific test sets, continuous evaluation, logging, and human calibration of automated scoring.
- paperEvaluating large language models trained on code ↗
HumanEval, and the formal pass@k estimator: the chance that at least one of k samples passes the tests. Not the same as passing k times in a row.
Sampling
How the model picks one token out of the distribution.
Temperature
The dial between predictable and creative.
Evaluation · Evals
How you know the system is actually working.
Regression Test
A test that catches a specific past failure coming back.