Physics of Context Windows

Context engineering with Dex Horthy

Topic 4 of 10

Published Jul 15, 2026 · Clip 28:15–33:22 (5:07)

The discussion turns to the 'physics' of context windows: Dex's tweet showing quality collapsing toward 1M tokens. He explains that longer context models aren't smarter — a model's intelligence drives its ability to attend across the window. He cites a 2025 study showing frontier LLMs reliably follow only ~150-250 instructions, and splits context engineering into information budget (rag-style selection) and instruction budget (too many or conflicting instructions degrade attention). He notes engineers now need intuition about attention dynamics and quadratic attention, learned mostly through debugging failures, not textbooks.