Transformer
The 2017 network architecture that all modern language models are built on.
A neural network architecture built around attention. Unlike earlier sequence models, it processes every token in parallel and can look directly at any other token, which is what made training on internet-scale data practical.
Introduced in the paper "Attention Is All You Need" (2017). The transformer block — attention plus a small feed-forward network, with residual connections and layernorm — is still the unit repeated dozens or hundreds of times to make a modern LLM.
Two consequences follow from the architecture and shape everything downstream. First, attention is quadratic in sequence length — doubling the window roughly quadruples that part of the compute, which is the real reason long context is expensive rather than free. Second, because the whole sequence is processed in parallel, the model has no built-in notion of order, so position has to be injected explicitly.
Why long context costs what it costs
A team assumes a 200k-token window makes retrieval unnecessary.
Pushing a whole repository into the window raises cost and latency superlinearly, and — per the long-context results — recall does not scale with it. The cheaper architecture is still to retrieve the right 5% and keep the window small.
Explaining the architecture in one sentence
Onboarding an engineer who has used the API but never read the paper.
"Every token produces a query and every token offers a key and a value; the output is a weighted sum of values, chosen by query-key similarity — and that happens in parallel at every layer." That is enough to reason about context, caching and cost.
- paperAttention is all you need ↗
The Transformer paper itself: sequence modelling built on attention instead of recurrence, and therefore parallelisable over the whole sequence.
- guideLLMs: what's a large language model? ↗
Encoder, decoder and decoder-only designs, self-attention, parameters and token-by-token generation, explained from the ground up.
- paperRoFormer: enhanced Transformer with rotary position embedding ↗
Rotary position embeddings, which encode position as a rotation and give self-attention relative-position structure. Behind most modern long-context extensions.
Attention · Self-attention
The mechanism that lets each token weigh every other token in the window.
LLM · Large Language Model
A very large model whose only job, at heart, is predicting the next token.
Neural Network
A stack of simple units whose weighted connections are learned from data.
Context Window
The hard ceiling on how much the model can see at once.