Sampling
How the model picks one token out of the distribution.
The rule that chooses the next token from the model’s predicted probabilities. Greedy sampling always takes the top token; top-p and top-k restrict the choices; temperature reshapes how peaked the distribution is.
Sampling is why two identical requests can give two different answers even with no randomness in the network itself.
Sampling rules are where determinism goes to die, and they interact with everything else: a temperature change invalidates your evaluation baseline, and decoding settings are part of the system you have to version alongside the prompt.
Sampling as a retry strategy
A code-generation step fails on the first attempt.
Do not raise the temperature and hope. Sample several candidates at a low temperature and run the tests against each — that is what pass@k measures, and it converts variance into a controlled retry mechanism rather than a quality gamble.
- paperEvaluating large language models trained on code ↗
HumanEval, and the formal pass@k estimator: the chance that at least one of k samples passes the tests. Not the same as passing k times in a row.
- guideIntroduction to large language models ↗
A plain-language official introduction to models, tokens and context. Teaching material rather than origin research — good for the definition, not for a claim about the frontier.
Temperature
The dial between predictable and creative.
Non-determinism
The same input can produce different output, and that is by design.
Token
The chunk of text the model actually sees — and the unit you are billed in.
Structured Output · JSON mode
Making the model return data your code can parse.