Interpretability
also called Explainability, XAI
Working out why the model did that.
The study of making model behaviour legible — from feature-level analysis inside networks to practical tools that attribute an output to its inputs.
For most teams the usable form is attribution: which retrieved passage drove which claim, which tool call led to which action. Mechanistic interpretability — reading features inside weights — is frontier research, not a product feature.
There is a gap between the research ambition and the practical use, and both are worth knowing. Feature-level work inside networks is not yet something a product team applies; attributing an output to its inputs, by contrast, is available now and is what makes most incidents explainable.
Why a reasoning trace is not an audit log
A model explains its reasoning and the explanation sounds convincing.
Tests on chain-of-thought faithfulness found models often use a hint without disclosing it, so the trace can omit the actual cause of the answer. Use it as a helpful signal and do not treat it as ground truth about why the output happened.
- blogReasoning models don't always say what they think ↗
Tests whether the visible chain of thought reflects the real cause of an answer. Often it does not, which caps how much you can trust a reasoning trace as an audit log.
- blogSleeper agents: training deceptive LLMs that persist through safety training ↗
Trigger-dependent backdoors that survived standard safety training — why "it passed the safety fine-tune" is not proof a bad behaviour is gone.