Drift
Quiet degradation as the world, or the model, changes under you.
The slow divergence between what a system was built for and what it now faces — changing user inputs, shifting data distributions, or a provider silently updating a model behind an unchanged name.
Drift is why point-in-time evaluation is not enough. Keep a standing eval set and run it on a schedule, and pin model versions where the provider allows it.
Drift comes from both directions and only one of them is under your control. Inputs change — new products, new customer language, seasonal patterns. And the system changes underneath you — a provider updates a model behind an unchanged name, or retires one you depend on. The first is detected by monitoring, the second by reading deprecation notices and pinning versions.
The silent model update
Quality drops with no change on your side and no deploy in the release log.
This is exactly what continuous evaluation on live traffic is for: a scored sample of production requests that would have flagged the shift on day one. Without it, the discovery happens through user complaints, weeks later and harder to attribute.
- docsDeprecations ↗
Model and endpoint retirement with notice periods and replacement mappings. The evidence that model pinning is a dependency you have to plan around.
- docsLangSmith evaluation ↗
Distinguishes offline evaluation on datasets from online monitoring of production traffic, and shows failing production traces being promoted into regression datasets.
Evaluation · Evals
How you know the system is actually working.
Regression Test
A test that catches a specific past failure coming back.
Observability
Being able to see what the system actually did.
Model Provider
The company or platform you call to run inference.