Quantization
Storing weights at lower precision to make a model cheaper to run.
Reducing the numerical precision of a model’s weights — from 16-bit floats to 8-bit or 4-bit integers — so it uses less memory and runs faster, usually with a small quality cost.
Quantization is what makes running a capable model on a single GPU (or a laptop) possible. The tradeoff is a measurable but often acceptable drop in accuracy.
The framing that helps in a procurement conversation: quantisation is a memory and throughput decision with a quality cost you should measure rather than assume. Eight-bit is usually indistinguishable on task accuracy; four-bit is where the interesting trade-offs live, and where your specific task may or may not survive.
Fitting the model on the hardware you have
A 70B model at 16-bit precision does not fit the available GPUs.
Four-bit post-training quantisation cuts the weight footprint by roughly four times and usually keeps general behaviour intact, at some cost on long reasoning chains. Evaluate on your own task before trusting the general benchmark claims.
- paperGPTQ: accurate post-training quantization for generative pre-trained transformers ↗
Post-training weight quantisation to low bit-width with limited reported accuracy loss — the method behind a lot of "runs on one GPU" claims.
- paperQLoRA: efficient finetuning of quantized LLMs ↗
Backpropagate through a frozen 4-bit quantised base model into LoRA adapters, which is what makes single-GPU fine-tuning of large models ordinary.
Open Weights · Open-source model
A model whose trained parameters you can download and run yourself.
Distillation
Training a small model to imitate a big one.
Parameters
How big the model is, counted in learned numbers.
Throughput
How many tokens per second the model generates.