Next-token Prediction
The single objective: guess what comes next, one token at a time.
The training task and generation loop of a language model — given everything so far, produce a probability distribution over the next token. Generation samples from that distribution, appends the result, and repeats.
This one mechanism, scaled up, produces translation, code, arithmetic and something that looks like reasoning. That it works at all is the central strange fact of the field.
This is the objective, the generation loop and the explanation for a whole class of failures. The model has no internal fact-check and no notion of "I do not know" unless training taught it to express one — because for most of its training, a confident plausible continuation scored exactly as well as a true one.
Hallucination as a scored incentive
An evaluation rewards correct answers and does not penalise confident errors.
Then guessing is the rational strategy, and the model will guess. Scoring abstention as a valid outcome — rather than only correctness — is the highest-leverage change available to most teams evaluating these systems.
- paperLanguage models are few-shot learners ↗
Introduces GPT-3 at 175B parameters and shows that scale alone produced in-context learning: performance from examples in the prompt, with no gradient updates.
- paperWhy language models hallucinate ↗
The statistical argument that training and grading regimes reward guessing over admitting uncertainty — so hallucination is an incentive problem, not a gap in knowledge.
- guideIntroduction to large language models ↗
A plain-language official introduction to models, tokens and context. Teaching material rather than origin research — good for the definition, not for a claim about the frontier.
LLM · Large Language Model
A very large model whose only job, at heart, is predicting the next token.
Token
The chunk of text the model actually sees — and the unit you are billed in.
Sampling
How the model picks one token out of the distribution.
Temperature
The dial between predictable and creative.
Faithfulness · Grounding
Sticking to the material it was given.