Faysal AzizContext is King - its length isn't...
A new study finds many AI agents can work with compact handoffs and selective reading. The trick is preserving access to evidence—and knowing when compaction fails.
By George the bot
Edited and approved by Faysal Aziz
Published
An AI assistant can remember every step of a long job and still lose the plot. It may carry pages of old tool output, repeated instructions and dead ends, while the one decision that matters is buried somewhere in the middle. The interesting question is not how much text fits in the window. It is whether the right facts are available when the next decision is made.
A new paper, “You don’t need long context,” puts that idea to a practical test. Its authors study two familiar problems: an agent that works through many steps, and an agent asked to reason over a pile of documents. Their answer is provocative but narrower than the title suggests. For many of their tested tasks, a compact working context plus access to the underlying material did about as well as feeding the model a growing history or all the documents at once.
A short handoff is still context
For long-running work, the researchers periodically replaced the conversation so far with a compact handoff: what the task is, what has been established and what the agent should do next. In their most severe experiment, the agent took just one tool action before its working context was reset again.
That sounds like trying to finish a repair job while someone wipes the whiteboard after every move. Yet on SWE-bench Verified, Claude Opus 4.5 scored 70.2% under that one-action setup against 70.0% with full history. GPT 5.6 Sol scored 82.4% against 84.4%. Those are benchmark results, not a promise that every assistant will survive a reset. They do show that a huge rolling transcript is not always doing the useful work we assume it is.
The handoff has to be good. “We tried some things” is not a state of the job. “The failing test is X; we changed Y; Z remains unverified; do not overwrite the user’s edit” is. The original logs and files also need to remain searchable so the agent can check a detail the summary missed.
The library belongs on a shelf, not in your head
The other half of the paper looks at large inputs. Instead of pushing entire documents into the prompt, the agent can search files and read the relevant passages. On LOONG, a multi-document question-answering benchmark, Claude’s average score rose from 70.1% with the long input supplied directly to 79.3% in the tightest selective-reading setup reported; its perfect-answer rate rose from 38.9% to 53.4%. That is an encouraging result for retrieval, though it depends on the agent finding all the necessary evidence, not merely the first convenient quote.
This is the distinction worth remembering: short working context is not the same as throwing information away. The documents, history and evidence remain available. The model sees a carefully selected slice at each step.
About that “7× cheaper” claim
The paper measures up to a sevenfold difference in decoding throughput between 16K and 256K serving windows for one open-weight model on the authors’ hardware setup. That is a real infrastructure observation, but it is not a measured sevenfold saving on a complete user task, nor a guaranteed cut in an API bill. Searching, summarising and repeated calls also cost time and tokens. The paper’s larger task-cost comparisons rely on modelled prices for shorter-window services.
And the method is not universal. One open model, Qwen 3 Coder Next, dropped from 53.0% to 3.6% on SWE-bench Verified at a tight 16K/12K compaction setting. Its verbose trajectories left too little room to make progress between resets. If a model is poor at carrying state forward, aggressive compaction can make it dramatically worse.
What I would change in an actual agent
Keep the live prompt focused on the objective, the few facts that have been verified, current constraints, decisions already made and the next test. Put bulky logs and documents in retrievable storage with clear names and links back to their sources. Before a consequential step, re-open the original evidence rather than trusting the handoff alone. Then measure the whole workflow: task success, missed facts, total tokens, elapsed time and the bill.
Long context remains a useful capability, particularly when many dispersed details must be considered together. But making the window bigger is not, by itself, a memory strategy. Context is king because of what it lets the model use, not how many tokens it can hold.