Neural Network
A stack of simple units whose weighted connections are learned from data.
A model made of layers of simple numerical units. Each unit takes inputs, multiplies them by learned weights, adds them up, and passes the result through a non-linear function. Stack enough of them and the whole thing approximates very complicated functions.
The biological naming is mostly a metaphor that misleads more than it helps. A neural network is a big differentiable function full of tunable numbers, and training is the search for numbers that make the output good.
Nothing in a network is a symbol or a rule. A "cat detector" is a direction in a high-dimensional space that happens to correlate with cats. That is why these systems degrade gracefully rather than crashing, and why they can also be confidently wrong in ways that feel inexplicable.
Weight count as a cost estimate
Choosing between a 7B and a 70B open-weights model for on-prem deployment.
Parameters are the memory bill: 70B at 16-bit precision is roughly 140GB of weights, before any activations or cache. That single multiplication usually decides the hardware conversation faster than any benchmark.
- guideLLMs: what's a large language model? ↗
Encoder, decoder and decoder-only designs, self-attention, parameters and token-by-token generation, explained from the ground up.
- paperLoRA: low-rank adaptation of large language models ↗
Freeze the pretrained weights, train small low-rank matrices beside them. The origin of parameter-efficient fine-tuning, and why adapters are cheap to ship and swap.
Deep Learning
Machine learning with many-layered neural networks — the technique behind modern AI.
Parameters
How big the model is, counted in learned numbers.
Weights · Parameters
The numbers inside the model that training moved.
Training
The expensive one-off process that sets the model’s parameters.