Observability
Being able to see what the system actually did.
Logging and inspecting requests, assembled prompts, model responses, tool calls, token counts, costs and outcomes, so behaviour can be understood after the fact and monitored continuously.
For AI systems this means logging the context, not just the request — an answer is only explicable alongside what the model was shown. Mind privacy: store what you need and redact the rest.
What to record, concretely: the assembled request, which tools were called with what, token counts and cost, latency per step, the outcome, and any user feedback. Without the assembled request in particular, most quality problems are uninvestigable.
Instrumenting before launch
A team is about to ship an agent to internal users.
The first complaint arrives within a day, and the question is always "what did it actually see". If the assembled context and tool calls are logged, the answer takes a minute; if not, the only recourse is asking the user to reproduce it.
- standardOpenTelemetry GenAI semantic conventions ↗
Agreed attribute names for spans, metrics and events covering GenAI clients, MCP and provider specifics — what makes traces comparable across vendors instead of bespoke.
- docsLangSmith observability ↗
A concrete example of what an LLM observability product records: production traces, dashboards, alerts and user feedback. First-party, so not an independent comparison of tools.
Tracing
Following one request through every step it took.
Prompt Versioning
Treating prompts as versioned, reviewable artifacts.
Evaluation · Evals
How you know the system is actually working.
Interpretability · Explainability
Working out why the model did that.