Pretraining
The first, most expensive pass: predict text at internet scale.
The initial training stage where a model learns from a huge general corpus by predicting the next token. This is where general capability, world knowledge and language fluency come from.
Pretraining is measured in thousands of GPUs, months of time and millions of dollars. Almost nobody outside the frontier labs does it for general-purpose models.
The Chinchilla correction is the detail worth carrying around: for a fixed compute budget, model size and the number of training tokens should grow together. Before it, models were systematically oversized and undertrained. After it, "how much data did it train on" became as relevant a question about a model as "how many parameters".
Two 70B models are not the same model
Comparing two open-weights models with identical parameter counts.
Ask for the token budget as well. A model trained on far more tokens can outperform a larger, less-trained sibling, which is why parameter count alone is a poor proxy for quality in model selection.
- paperTraining compute-optimal large language models ↗
The Chinchilla correction: for a fixed compute budget, model size and training tokens should grow together. Most earlier large models were, by this measure, over-sized and under-trained.
- paperLanguage models are few-shot learners ↗
Introduces GPT-3 at 175B parameters and shows that scale alone produced in-context learning: performance from examples in the prompt, with no gradient updates.
- paperLlama 2: open foundation and fine-tuned chat models ↗
A released family from 7B to 70B plus chat variants, with the fine-tuning and safety work documented. A primary example of published weights — not a ruling on what "open source" should mean.
Training
The expensive one-off process that sets the model’s parameters.
Foundation Model · Base model
A large model trained broadly, meant to be adapted to many downstream tasks.
Next-token Prediction
The single objective: guess what comes next, one token at a time.
Fine-tuning
Extra training on your own examples to change how the model behaves.