Perplexity
How surprised the model is by the text — a language-modelling score.
A measure of how well a model predicts a body of text: lower means the text is more expected. Historically central to language modelling; weakly related to whether output is useful.
Good for comparing models on a fixed corpus, poor as a proxy for task quality. Do not use perplexity to decide whether your assistant got better.
It remains the right metric for training and for measuring how well a model models a distribution, and the wrong metric for whether an output is useful. A low perplexity means the text was predictable, which is a statement about surprise and not about truth, relevance or safety.
Why perplexity stopped being the headline
Explaining to a non-specialist why models are compared on tasks now.
A model can be excellent at predicting the next token and still fabricate confidently, because plausibility is exactly what low perplexity measures. Task-based evaluation replaced it because it asks whether the output is right, not whether it was expected.
- paperLanguage models are few-shot learners ↗
Introduces GPT-3 at 175B parameters and shows that scale alone produced in-context learning: performance from examples in the prompt, with no gradient updates.
- paperTraining compute-optimal large language models ↗
The Chinchilla correction: for a fixed compute budget, model size and training tokens should grow together. Most earlier large models were, by this measure, over-sized and under-trained.