Attention
also called Self-attention
The mechanism that lets each token weigh every other token in the window.
A learned lookup where each token produces a query and every token offers a key and a value. Matching a query against all keys produces weights, and the output is the weighted sum of the values — so the model can pull in whatever earlier context it needs.
Attention is also the reason context is expensive. Pairwise comparisons across the whole sequence scale quadratically with length, which is why the industry is so focused on making attention cheaper.
The practical reading of attention for anyone building with these systems is that context is a budget, not a container. Every token you add competes for the same fixed allocation, which is why a wall of retrieved documents can outperform nothing at all yet still underperform three good paragraphs.
Padding the window makes it worse
A RAG pipeline stuffs twenty chunks into every prompt to be safe.
Quality drops even though every relevant chunk is present. Reduce to the five highest-scoring chunks after reranking and hold the rest behind a tool the agent can call. Less context, better answers — the counter-intuitive result that follows directly from how attention allocates.
- paperLost in the middle: how language models use long contexts ↗
Measured U-shaped performance: relevant information in the middle of a long context is used less reliably than at the beginning or end. Order your context deliberately.
- blogEffective context engineering for AI agents ↗
Defines context engineering as curating everything supplied at inference, names context rot, and covers just-in-time retrieval, compaction and keeping the window high-signal.
Transformer
The 2017 network architecture that all modern language models are built on.
Attention Budget
The finite share of the model’s focus each token gets.
Attention Degradation · Lost in the middle
Recall drops for content buried in a large window.
Context Window
The hard ceiling on how much the model can see at once.