Your AI Remembers Everything — Until the Facts Start Changing: Why the Best Models Score 13% on StateBench

Your AI Remembers Everything — Until the Facts Start Changing: Why the Best Models Score 13% on StateBench

Ask your favourite AI to find one fact in a 100-page document and it will nail it. Ask it to follow that fact as it changes — the price updated three times, the owner swapped twice — and the longer the story gets, the more the model quietly loses the thread. It does not forget. It loses track.

A new paper from the Allen Institute for AI (Ai2) and NYU puts a number on it: their benchmark, StateBench, stretches state tracking from 8 to 64,000 assignments — and on the longest, hardest tasks, four of the best frontier models scored 13% or lower. The paper is "A Fine-Grained Hierarchy and Benchmark for State Tracking in Language Models" by Jackson Petty, William Merrill and Ashish Sabharwal (arXiv:2611.03094, 2 November 2026).

What the researchers did

The paper starts with a distinction: finding a fact is not the same as tracking a state. Long-context benchmarks test retrieval — finding the needle in the haystack. State tracking asks the model to carry variables through a long chain of updates and report the final state. That is what code evaluation, legal review and agent work actually require.

To measure it, the researchers built a four-level hierarchy, grounded in the Krohn–Rhodes decomposition — the algebra of how small operations combine into complex state machines:

Level 1 — r-trivial: assignments never change: a variable gets a value once, like a fixed badge number.

Level 2 — aperiodic: assignments are mutable: a later value overwrites an earlier one — red, then blue, means blue.

Level 3 — soluble: assignments are self-referential: an update depends on the variable's own current state, like x = x + 5.

Level 4 — general finite: assignments by reference, including swaps: a variable can take another's current value, so an early mistake corrupts everything downstream.

StateBench holds the number of variables, values and assignment positions constant across all four levels — only the update rules change. So a worse score at a higher level means the type of tracking broke, not the length.

Then came the theory: the class of problems transformers are thought to length-generalise on does not even contain all aperiodic tracking — level 2, the "just remember the latest value" case. By contrast, Gated DeltaNet (GDN), a recurrent architecture with positive eigenvalues, expresses all aperiodic problems, and the regular languages recognised by certain transformers are precisely all soluble ones — level 3 is the transformer's ceiling.

Finally, the theory became StateBench: 68,000 sequences, each pairing one correct final state with four incorrect candidates, spanning 8 to 64,000 assignments. The team trained six 275-million-parameter models from scratch — two transformers, two hybrids, two GDN variants — on sequences up to 256 assignments, then tested them up to 64,000 long: 250 times longer than training. Results largely matched the theory. And when pretrained models were evaluated — Olmo 3, the Olmo Hybrid, plus closed frontier models — none robustly solves even soluble tracking.

Why it matters

First, it exposes what long-context scores hide: a model can ace a 1-million-token retrieval test and still collapse on a 64,000-assignment tracking task. Long context is about accepting a long input; state tracking is about preserving the state inside it. If you choose a model for work with changing facts, the context-window number is not the one that matters. The state-tracking number is.

Second, it is a warning about the models you already use. Ai2 reports Gemini 3.8 Flash, Claude Opus 5.5, GPT-6 Sol and GPT-6 Luna all did well on the easy levels but fell sharply on soluble and general-finite problems as inputs lengthened, reaching 13% or lower on the longest cases. That is not a gap between models — it is all of them hitting the same wall. When your agent silently returns the wrong final state after a long session, this is very likely why.

Third, the paper points at the fix: recurrent architectures like GDN can express classes of tracking that transformers cannot length-generalise on, and the hybrid results back that up. Expect the next wave of "long-context" models to be hybrids — attention plus a recurrent state-tracking layer — judged by StateBench, not retrieval tests. Dataset and code are public (Apache-2.0).

How YOU can use it

You do not need a GPU cluster to use this paper: its core idea is a test you can run on any AI assistant in five minutes, plus a habit that protects you from its failure mode.

Step 1: Build a 2-minute state-tracking test. Write 20–30 chained updates: "Warehouse A holds 40 units. Move 12 to B. B's stock doubles. Swap A and C…" Ask for the final stock of each warehouse, then run the same test at 200 updates. If accuracy collapses as the chain grows, you have found your model's real working limit — probably far shorter than its advertised context window.

Step 2: Make the model show its ledger. Before asking for a conclusion, ask the model to list every variable's current value in a table. A model that can recite the ledger is tracking; one that jumps straight to the answer is guessing.

Step 3: Keep the state outside the model. For any real workflow — a project plan, a budget, a refactor — keep a short state file (a markdown table of variables and current values) that you own. Paste it in at the start of each session; at the end, ask the model to output the updated table and save it. The file is your memory; the model is your calculator. The model's memory of changing facts is the weak link, so do not make it the only copy.

Step 4: Test before you deploy. If you are picking a model for an agent that will run for hours, run a StateBench-style check first: feed it a long, changing scenario from your own domain and score the final state. The data and code are public — but even the 5-minute version from Step 1 tells you more than any leaderboard about whether that model survives your workload.

ILLUSTRATIVE sketch — not code from the paper:

# A 5-minute StateBench-style test for any AI assistant
# (Illustrative pattern inspired by the paper — not the paper's code.)
# Level 1 — r-trivial: value never changes
"Bin A = 40. Bin A is mentioned again later. What is Bin A?"
# Level 2 — aperiodic: latest value wins
"Bin A = 40. Bin A = 25. Bin A = 60. What is Bin A now?"
# Level 3 — soluble: self-referential updates
"Bin A = 10. Add 5 to Bin A. Double Bin A. Subtract 3 from Bin A.
 What is Bin A now? (Answer: 27)"
# Level 4 — general finite: swaps between variables
"Bin A = 10, Bin B = 20. Swap A and B. Add B's value to A.
 What are A and B now? (Answer: A = 30, B = 10)"
# The test: run each level at 20 updates, then at 200.
# Score the final state, not the reasoning.
# If level 3 or 4 collapses as the chain grows, that model
# will quietly lose track in your long sessions too.

One guardrail: do not "fix" a failing model by stuffing more context into the prompt. The failure is not about how much the model can hold — it is about how it carries change. More context makes the chain longer, which is exactly what makes tracking worse. Shorter chains, explicit ledgers and a state file you own beat a bigger window.

Key takeaways

• Finding a fact in a long document is easy for modern AI. Tracking a fact as it changes is not — and StateBench isolates the difference.
• The hierarchy runs r-trivial (fixed values) → aperiodic (overwrites) → soluble (self-referential) → general finite (swaps), with only the update rules changing.
• Theory says transformers cannot length-generalise on even all aperiodic tracking, while Gated DeltaNet with positive eigenvalues can — and certain transformers top out at the soluble level.
• On StateBench's longest, hardest tasks (up to 64,000 assignments), four frontier models — Gemini 3.8 Flash, Claude Opus 5.5, GPT-6 Sol, GPT-6 Luna — scored 13% or lower, and no pretrained model tested robustly solves even soluble tracking.
• The practical fix is yours: test with chained updates, demand a visible ledger, and keep the state in a file you own.

Read the original paper: A Fine-Grained Hierarchy and Benchmark for State Tracking in Language Models (arXiv:2611.03094) by Jackson Petty, William Merrill and Ashish Sabharwal. Link: https://arxiv.org/abs/2611.03094