Attention Budget
The finite share of the model’s focus each token gets.
The informal limit on how much of the window the model can actually track well. Every token competes for attention, so adding context dilutes the rest rather than simply adding information.
This is why a prompt with one clear instruction beats a prompt with twenty. Context is not storage — it is a competition for focus.
Treating attention as a budget rather than a container explains a result that otherwise looks paradoxical: adding a correct document can reduce answer quality, because it competes with the documents that already sufficed. Retrieval quality means precision as well as recall.
Fewer chunks, better answers
A pipeline retrieves twenty chunks and quality is mediocre.
Rerank and keep the best five. Nothing was added; the two relevant paragraphs simply stopped competing with eighteen irrelevant ones. This is the cheapest quality improvement available to most retrieval systems.
- blogEffective context engineering for AI agents ↗
Defines context engineering as curating everything supplied at inference, names context rot, and covers just-in-time retrieval, compaction and keeping the window high-signal.
- paperLost in the middle: how language models use long contexts ↗
Measured U-shaped performance: relevant information in the middle of a long context is used less reliably than at the beginning or end. Order your context deliberately.
Attention · Self-attention
The mechanism that lets each token weigh every other token in the window.
Attention Degradation · Lost in the middle
Recall drops for content buried in a large window.
Smart Zone
The window size where the model is still sharp.
Context Window
The hard ceiling on how much the model can see at once.
Context Rot
A long session slowly getting worse.