Your AI Agent Doesn't Need a 200K Memory — NVIDIA Proves 8K Is Enough (and 7x Cheaper)

Your AI agent remembers everything. Every tool call, every error message, every retry — it all stays in the conversation, and the context window keeps growing. It feels thorough. According to new research from NVIDIA, it is also mostly a waste of money.

A paper published on 3 November 2026 — "You don't need long context" by Peter Belcak and colleagues at NVIDIA and ETH Zurich (arXiv:2611.05306) — shows that a short, compacted context of around 8K tokens matches or beats a full 200K-token history on real coding work, while costing up to 7x less.

The reason is simple: in a long conversation, later tokens cost more than early ones — up to 7x more. Every extra thousand tokens of history you drag along makes every new token the model generates more expensive. Long context is not just slow; it is a tax you pay on every single step your agent takes.

What the researchers did

Belcak and his co-authors tested coding agents on SWE-bench, the standard benchmark for real-world software engineering tasks. They compared agents running with their full conversation history against agents running on a short, compacted summary of the same work. On SWE-bench, the compact-context agents matched — and in some cases beat — the agents carrying the full 200K-token history.

That is the result that matters: you do not need the transcript. You need the state.

The fix: a 4-part state summary

Instead of feeding the model the whole history, the researchers compact the conversation into a short, structured summary of the current state — four parts that capture where the task stands — and let the agent keep working from that summary alone. The summary is refreshed as the work progresses, so the agent never loses track, but it never has to re-read everything either.

The result is an agent that works from roughly 8K tokens of context instead of 200K, at roughly a seventh of the cost — and performs just as well.

How YOU can use it

You do not need to wait for a framework to ship this. You can apply the same idea to your own agent workflows today:

• Cap the working context. Aim for a compact working set of around 8K tokens instead of letting history grow to 200K.
• Keep a rolling state summary. After each meaningful step, write down where the task stands — the goal, what has been tried, what was learned, and what happens next — and work from that summary instead of the raw transcript.
• Refresh, don't accumulate. Update the summary as you go so it stays current, and let old raw history drop away.
• Keep the raw log somewhere cheap. If you need the full history for debugging, store it outside the context window and bring back only the pieces that matter.

Smaller context is not a limitation. It is the feature: cheaper, faster, and — as NVIDIA's results show — just as capable.

The bottom line

Long context has been sold as the answer to smarter agents. This paper says the opposite: what makes an agent smart is not how much it remembers, but how well it remembers what matters. An 8K state summary can do the job of a 200K transcript — for a seventh of the price.

Paper: "You don't need long context" — Peter Belcak et al. (NVIDIA, ETH Zurich), arXiv:2611.05306, 3 November 2026. Read it at arxiv.org/abs/2611.05306.