Your AI Agent Doesn't Need a 200K Memory — NVIDIA Proves 8K Is Enough (and 7x Cheaper)

The AI industry is locked in a context-window arms race. Every model launch boasts a bigger number — 128K, 1M, 2M tokens — as if remembering everything were the same as understanding everything. A new paper from NVIDIA and ETH Zurich asks the heretical question: what if most of that memory is waste? Their answer, tested on frontier models doing real software engineering, is that a tiny, constantly refreshed memory works just as well — sometimes better — and costs up to 7 times less.

The paper is "You don't need long context" by Peter Belcak and colleagues (arXiv:2611.05306, 3 November 2026). It is not a theory piece. The researchers ran Claude Opus 4.5 and GPT 5.6 Sol on SWE-bench Verified and Terminal-Bench 2.1 — the gold-standard benchmarks for agents that actually write code and run commands — and measured what happens when you stop feeding the model its entire history.

What the researchers did

First, they measured the tax nobody talks about. In a standard transformer, generating each new token means attending over the entire KV cache of everything that came before. The cost is not flat: on frontier open-weight models, generating a token at 128K context costs 3.6 to 7 times more than generating one at 8K. And long agent sessions balloon fast — every tool call, every command output, every dead end gets appended, and contexts in the study grew to 130K–200K tokens. You are paying premium prices for your agent to re-read its own mistakes.

Then they tested three ways of managing that memory:

1. Full-history accumulation (FHA). The default. Everything stays in the prompt forever. This is how most agent frameworks work today.

2. End-of-context compaction (EoCC). The agent works inside a bounded window — 24K to 48K tokens. When the token count crosses a threshold, an internal compaction pass summarises the history, keeps the essential findings and constraints, and throws away the raw logs.

3. Single-action compaction (SAC). The radical one. The model takes exactly one action per context. After receiving the environment's feedback, it writes out a revised state summary and starts a fresh invocation with no transcript at all.

Behind SAC sits the paper's big idea: the Semantic State Space Model (SSSM). Instead of a hidden vector nobody can read, the agent's memory is a short plain-language state — its goals and roadmap, the facts it has verified, the paths that failed, and its current sub-goal. After each action and observation, the state is updated: S_t+1 = U(S_t, A_t, O_t). Because the state is written in words, a human can read it, audit it, and checkpoint it. The KV cache stays bounded at roughly 8K tokens instead of growing without limit.

The results are the part that should change how you build agents. With Claude Opus 4.5, capping context at 24K with compaction at 18K did not just save money — it improved SWE-bench Verified accuracy from 70.0% to 74.8%, and Terminal-Bench from 47.2% to 52.8%, at roughly 3.8x lower generation cost. Under the radical single-action regime, Opus still scored 70.2% on SWE-bench and 51.8% on Terminal-Bench — matching or beating its own full-history baseline — with a median peak context of just 8K tokens and about 5.5x lower cost. GPT 5.6 Sol showed the same pattern under compaction: 84.4% to 88.8% on SWE-bench. Peak context collapsed from 200K to 16–22K — an 89% reduction in KV cache.

There is one honest caveat, and the paper does not hide it: under the most extreme single-action setting, GPT 5.6 Sol dipped slightly — 82.4% on SWE-bench versus 84.4% with full history. The radical mode is not free for every model. But the middle path, compaction inside a bounded window, was the sweet spot for both models tested: cheaper and more accurate.

Why it matters

First, the 7x tax is real money. API pricing charges a flat rate per token, but serving a token at 128K context costs up to 7x more compute than serving one at 8K. Long agent sessions are exactly where the bill explodes — and they are also where the model is re-reading the least useful text.

Second, shorter context can be better, not just cheaper. This is the counterintuitive finding. Pruning the noise improved accuracy: with less stale output competing for attention, the model suffered less from the "lost in the middle" effect and reasoned more cleanly. Memory is not just a cost; uncurated memory is a distraction.

Third, auditable agents become possible. A plain-language state is something you can actually inspect mid-run. You can see what your agent believes, what it has verified, and which paths it has already ruled out — instead of trusting a 200K-token transcript nobody will ever read. That matters for debugging, for guardrails, and for any regulated setting where "the model did it" is not an acceptable explanation.

Finally, it questions the arms race itself. If frontier models solve long-horizon coding tasks with an 8K working memory, the race to 1M-token windows may be solving the wrong problem. The better investment might be smarter compaction, not a bigger bucket.

How YOU can use it

You do not need to wait for a new model or framework. The paper's recipe translates directly into how you prompt and configure the agents you already run:

Step 1: Set a context budget — and enforce it. Pick a cap the way the paper did: a 24K working window with compaction triggered at 18K. In practice, that means configuring your agent harness to summarise and restart the conversation when the token count crosses your threshold, instead of letting it grow until the model degrades or the bill shocks you.

Step 2: Give your agent a state summary, not a transcript. The SSSM state has four parts — copy this template into your agent's system prompt or compaction step:

WORKING STATE (update after every action):
1. GOAL: what I am trying to achieve, and the plan in 3-5 steps
2. VERIFIED FACTS: file paths, test results, and hypotheses already confirmed
3. DEAD ENDS: approaches tried and ruled out — do not retry these
4. NEXT STEP: the single immediate action to take

Step 3: Compact on a trigger, not on a schedule. When the context crosses your threshold, run a compaction pass with a prompt like this:

Summarise this session into the WORKING STATE format above.
Keep every verified fact, file path, and failed approach.
Drop all raw command output, stack traces, and intermediate reasoning.
The summary must let a fresh model continue the task with no other history.

Step 4: Prune tool output at the source. Most context bloat is not conversation — it is tool output. Cap command output to the last 50 lines, keep only the verdict of a test run (pass/fail plus the failing assertion), and never paste a full log file into the prompt. The paper's accuracy gains came largely from removing exactly this noise.

Step 5: Measure tokens per task, before and after. Pick one recurring agent task, log total tokens used with full history, then log it again with compaction at 18K. The paper's numbers say to expect roughly 4–5x less generation cost with accuracy flat or better — check it on your own workload before rolling it out everywhere.

The sketch below shows the shape of the loop — an illustrative pattern inspired by the paper, not code from it:

# Compacting agent loop (ILLUSTRATIVE sketch — not the paper's code.
# The paper's finding: bounded context + semantic state keeps accuracy
# while cutting generation cost ~4-5x.)
def run_task(task, budget=18000, window=24000):
    state = "GOAL: " + task + "\nVERIFIED FACTS: none\nDEAD ENDS: none\nNEXT STEP: start"
    while not done(state):
        action = agent.act(state)          # fresh, short context each turn
        observation = env.run(action)
        state = agent.compact(state, action, observation)  # S_{t+1} = U(S_t, A_t, O_t)
        if tokens(state) > budget:
            state = agent.summarise(state) # drop raw logs, keep verified facts
    return state

Key takeaways

• Later tokens cost more: generating a token at 128K context can cost 3.6–7x what it costs at 8K, because every token attends over the whole KV cache.
• Frequent semantic compaction works: frontier models kept — and sometimes improved — their accuracy on SWE-bench Verified and Terminal-Bench 2.1 with short, compacted contexts.
• Claude Opus 4.5 went from 70.0% to 74.8% on SWE-bench Verified with a 24K window compacted at 18K, at ~3.8x lower cost; even one-action-per-context held at 70.2% with an 8K median peak context.
• The Semantic State Space Model replaces an unreadable transcript with a short plain-language state — goals, verified facts, dead ends, next step — that humans can audit and checkpoint.
• The honest limit: the most extreme single-action mode cost GPT 5.6 Sol a couple of points, so bounded-window compaction — not the radical extreme — is the practical default.
• Your move today: cap your agent's context at ~24K, compact at ~18K with a four-part state summary, prune raw tool output, and measure tokens per task.

Read the original paper: You don't need long context (arXiv:2611.05306) by Peter Belcak, Saurav Muralidharan, Jindong Jiang, Yonggan Fu, Wonmin Byeon, Hanrong Ye, Ryan Angilly, Ali Taghibakhshi, Shih-Yang Liu, Siru Ouyang, Greg Heinrich and Pavlo Molchanov. Link: https://arxiv.org/abs/2611.05306