Mixture of Experts
also called MoE, Sparse model
A big model that only uses part of itself for each token.
An architecture where the network contains many expert sub-networks and a router sends each token to a small subset. The model has a large total parameter count but activates only a fraction per token, so inference is cheaper than a dense model of the same size.
This is why parameter counts became misleading as a quality proxy: a mixture-of-experts model with a large total may run with the cost of a much smaller dense model. Ask which number is being quoted — total or active parameters — before comparing two models.
The operational consequence is that memory and compute decouple: you need to hold all the experts in memory but only compute with a few per token. That shapes the hardware conversation, and it is why these models are attractive to serve at volume and awkward at low utilisation.
Comparing two models fairly
A large mixture-of-experts model and a smaller dense model are being compared.
Compare active parameters and measured throughput on your workload, not the headline total. The mixture model may be the cheaper option per token while needing more memory to hold — which matters if you are paying for provisioned capacity.
- paperSwitch Transformers: scaling to trillion parameter models with simple and efficient sparsity ↗
Sparse mixture-of-experts routing: each token activates only some parameters, so a large total parameter count does not mean paying for all of it per token.
- paperMistral 7B ↗
Grouped-query attention and sliding-window attention in a single model write-up: the concrete engineering behind cheaper inference and bounded attention.
Parameters
How big the model is, counted in learned numbers.
Weights · Parameters
The numbers inside the model that training moved.
Quantization
Storing weights at lower precision to make a model cheaper to run.
Throughput
How many tokens per second the model generates.
Model Provider
The company or platform you call to run inference.