Parameters
How big the model is, counted in learned numbers.
The total count of learnable values in a model. A 70B model has roughly seventy billion. More parameters generally means more capacity and more cost.
Parameter count is a rough proxy for capability, not a measure of it. Training data, architecture and post-training often matter more than raw size.
Parameter count is a rough capacity measure and a concrete memory cost, and it is a poor quality measure on its own. Which is why model selection conversations go better when they start from your eval results and work backwards to size, rather than starting from the biggest number available.
The bigger model that was worse
A larger model is chosen for a classification task because it "should be smarter".
It is slower, several times more expensive, and often no more accurate on a task that is really about following a label convention. Run both through the eval set; the smaller model usually wins on cost-adjusted quality, and the numbers settle the argument.
- paperScaling laws for neural language models ↗
Empirical power laws relating loss to parameters, data and compute — the justification for the "bigger is better" era of training.
- paperSwitch Transformers: scaling to trillion parameter models with simple and efficient sparsity ↗
Sparse mixture-of-experts routing: each token activates only some parameters, so a large total parameter count does not mean paying for all of it per token.
Weights · Parameters
The numbers inside the model that training moved.
Foundation Model · Base model
A large model trained broadly, meant to be adapted to many downstream tasks.
Quantization
Storing weights at lower precision to make a model cheaper to run.
Benchmark
A shared public test used to compare models.