Training
The expensive one-off process that sets the model’s parameters.
Adjusting a model’s parameters so its predictions get better on a training objective. In training the weights move; the arithmetic of the model itself stays fixed.
Training is separated in time from use. Nothing you type changes the weights — that only happens in a training run. This distinction is the source of most confusion about "learning from our conversation".
Two things are true at once and are worth stating separately. Training is where all the capability comes from, and it is also the part almost nobody using these systems will ever do. The realistic division of labour for a product team is: read about training so you understand the constraints you are inheriting, then spend your effort on the parts you control.
The scaling decision you inherit
A team wants their model to be better at their domain and asks about "retraining it".
They almost never want retraining, which would cost millions and lose the reasoning ability the base model already has. They want retrieval for facts and a small fine-tune for format and tone. Same outcome, four orders of magnitude cheaper, and reversible.
Saying the model "learned" from your chat session. Nothing was trained.
- paperScaling laws for neural language models ↗
Empirical power laws relating loss to parameters, data and compute — the justification for the "bigger is better" era of training.
- paperTraining compute-optimal large language models ↗
The Chinchilla correction: for a fixed compute budget, model size and training tokens should grow together. Most earlier large models were, by this measure, over-sized and under-trained.
Pretraining
The first, most expensive pass: predict text at internet scale.
Fine-tuning
Extra training on your own examples to change how the model behaves.
Inference
Using a trained model — the part you pay for on every request.
Parameters
How big the model is, counted in learned numbers.
Weights · Parameters
The numbers inside the model that training moved.