• Bottom line: Wait. Frontier models reliably recall the latest value of a variable but not dependent updates carried across long documents.
• The deciding number: accuracy of 13% or lower on the longest general finite problems for all four frontier models tested.
• Pretrained open models stayed at chance on the hardest level, but models trained directly on the task show the failure is not purely architectural for Gated DeltaNet and hybrids.
• Format changes the result: 275M hybrid and Gated DeltaNet models scored near ceiling on RULER-style problems but near chance when the same problems were framed as spreadsheet edits.
• StateBench assumes a fixed set of variables, so it does not yet measure the case where new entities appear mid-conversation.

The bottom line: Wait. Ai2's StateBench, described in a November 3, 2026 post, finds that four frontier models cope with simple variable updates over long documents but collapse on the hardest tier, with accuracy at 13% or lower on the longest general finite problems. We could not reproduce the runs; everything below comes from Ai2's own write-up and the dataset listing it links.

What the benchmark tests

StateBench has four levels, each permitting a more complex update to a variable: immutable assignment, reassignment where the latest value wins, self-referential updates such as x = x + 5, and assignment by reference such as swaps. Ai2 builds all four from one general sequence, so variable count and distinct values stay fixed and only the update type changes. That control is the benchmark's main strength.

Frontier models break where updates depend on each other

Gemini 3.8 Flash, Claude Opus 5.5, GPT-6 Sol and GPT-6 Luna all handled the two easiest levels well, according to Ai2. On soluble and general finite problems, accuracy fell sharply as length grew, reaching 13% or lower on the longest general finite problems. The authors suggest sensitivity plays a part: at harder levels more steps affect the answer, so one missed update is more likely to flip it.

Training, not only architecture, drives the gap

Models were trained on sequences of up to 256 assignments and tested on up to 64,000, roughly 250 times longer. Gated DeltaNet and hybrid models generalized almost perfectly on the first three levels, while transformers degraded on longer sequences and never learned the general finite level. Ai2 reads the gap between pretrained and directly trained results as pointing to training, not an architectural ceiling, for Gated DeltaNet and hybrids.

Prompt format can swing the score

Among pretrained models, most did well on the easiest level, lost accuracy on reassignment as inputs grew, and stayed at chance on the hardest level. The 275M hybrid and Gated DeltaNet models scored near ceiling on RULER-style easy problems but near chance when the same problems were framed as spreadsheet edits. A single-format result should therefore not be read as a general capability claim.

What we could not verify and what is missing

We did not rerun StateBench, and the dataset card we could read lists only metadata: tabular and text, English, in the 10K to 100K size range, with no license or task description visible. Ai2 also says the benchmark assumes a fixed variable set, while real conversations introduce new entities. Its proof about Gated DeltaNet holds only under fixed-precision arithmetic with rounding.

The bottom line: Wait

Wait before building long-document workflows that need a model to carry dependent updates, such as ledgers, inventories or multi-party edits, across tens of thousands of steps. The deciding number is 13% or lower accuracy for all four frontier models on the longest general finite problems. Where updates are simple overwrites, the reported results are more encouraging. Test your own sequences at production length, and treat the hybrid-architecture results as research findings, not shipping options, until independent replication appears.