Temperature
The dial between predictable and creative.
A scaling factor on the model’s output probabilities before sampling. Low temperature makes the model favour the most likely token; high temperature flattens the distribution and produces more variety and more nonsense.
Rough guide: 0 for extraction, classification and code; 0.2–0.5 for analysis; 0.7–1.0 for brainstorming and writing. High temperature for a task with one right answer is a self-inflicted wound.
Two operational rules. For anything extracted, classified or parsed, run as close to greedy as the API allows — you want the same input to give the same output. For open-ended writing, raise it and accept that you now need sampling in your evaluation to tell signal from noise.
The temperature that broke the pipeline
One field in a JSON response occasionally comes back as a sentence.
The schema was described in prose and the temperature was tuned up months earlier for a different feature. Schema-constrained output plus a near-zero temperature removes the entire failure class rather than reducing its frequency.
- guideIntroduction to large language models ↗
A plain-language official introduction to models, tokens and context. Teaching material rather than origin research — good for the definition, not for a claim about the frontier.
- docsEvaluation best practices ↗
Evals as structured tests: task-specific test sets, continuous evaluation, logging, and human calibration of automated scoring.
Sampling
How the model picks one token out of the distribution.
Non-determinism
The same input can produce different output, and that is by design.
Prompt Engineering
The craft of writing instructions a model reliably follows.
Structured Output · JSON mode
Making the model return data your code can parse.