Online Eval
also called Production evaluation, A/B test
Measuring with real users, on real traffic.
Evaluating in production via A/B tests, held-out traffic or scored samples of live requests. The only way to know whether the offline number translates into user value.
Offline evals catch regressions cheaply; online tests reveal what actually matters — task completion, retention, escalation rate. Choose metrics a non-technical stakeholder can argue about.
Offline evaluation tells you the change is not worse on the cases you thought of; online evaluation tells you whether it helps the people you did not think of. Systems with both move fast and safely, because a risky idea can go to a small slice of traffic and be answered with evidence in a day.
Shadowing before switching
A new prompt looks better offline and nobody trusts the eval set.
Run it alongside the current one and score both on a sample of live traffic. Comparing on real distribution catches the cases the offline set never contained, and it does so before any user is affected by the switch.
- docsLangSmith evaluation ↗
Distinguishes offline evaluation on datasets from online monitoring of production traffic, and shows failing production traces being promoted into regression datasets.
- docsEvaluation best practices ↗
Evals as structured tests: task-specific test sets, continuous evaluation, logging, and human calibration of automated scoring.
Evaluation · Evals
How you know the system is actually working.
Observability
Being able to see what the system actually did.
Human Eval · Human review
People scoring the output — the ground truth everything else is calibrated against.
ROI · Return on investment
Did it save more than it cost?