After the Transformer.
Keep a little attention, replace the rest with something whose memory does not grow.
This volume describes the field as of October 2026. Claims are marked established (replicated or long-standing), reported (from a single lab or report, not independently reproduced) or contested (actively disputed). A Transformer’s KV cache grows with every token (plate XXIII.2). State-space models such as Mamba (Gu and Dao, 2023) and linear-attention variants instead carry a fixed-size state from token to token, so memory and per-token cost do not grow with context — at the price of weaker exact recall of distant tokens. Hybrids keep a few attention layers for recall among many state-space or linear layers: NVIDIA’s Nemotron-H uses self-attention in 4 of 52 layers of its 8B model and reported comparable accuracy with up to 3× faster inference (reported), and several 2025–2026 open models adopted similar mixes (established as a design in use).
A different departure is diffusion language models, which generate a block of tokens in parallel by iterative refinement rather than one at a time (plate XXIX.5); Google and Inception reported throughputs above a thousand tokens per second for such models in 2025–2026, with quality behind comparable autoregressive models on hard benchmarks (reported). The workshop compares the per-sequence memory of a Transformer, a hybrid and a pure state-space model of Llama 3 8B’s width as context grows.
Below: memory per sequence against context length for the three designs; readouts at the chosen length.
- Explain why alternatives to attention are sought.
- Describe state-space models and hybrids.
- Compare memory growth with context.
- Separate reported from established claims.
Fixed memory, imperfect recall
At a million tokens of context the Transformer’s cache for one sequence is about eight times the hybrid’s (8.0× here), and the state-space model’s memory does not grow at all; below about 128 tokens the state-space model’s fixed state is actually larger than the Transformer’s cache. That is the case for hybrids at long context: most of the memory saving, with a few attention layers kept for what state-space layers do poorly — copying and exact retrieval from far back, where Transformers are measurably better (established on synthetic tasks).
Whether any non-attention design displaces the Transformer at the frontier is contested: hybrids are deployed in open models and reportedly in some production systems, diffusion language models are fast but have not matched the best autoregressive models in quality, and every proposal has had to match attention’s ease of training at scale. The safe summary for October 2026 is that the Transformer remains the backbone, increasingly with cheaper layers mixed in.
Terms
- State-space model
- A sequence layer carrying a fixed-size state from token to token.
- Hybrid architecture
- A few attention layers among many cheaper sequence layers.
- Diffusion language model
- A model generating a block of tokens in parallel by refinement.
- Recall
- Retrieving exact information from far back in the context.
Sources
- Gu and Dao, “Mamba: Linear-Time Sequence Modeling with Selective State Spaces”, arXiv:2312.00752, 2023.
- NVIDIA, “Nemotron-H”, arXiv:2504.03624, 2025.
- Lieber et al., “Jamba: A Hybrid Transformer-Mamba Language Model”, arXiv:2403.19887, 2024.
- Nie et al., “Large Language Diffusion Models” (LLaDA), arXiv:2502.09992, 2025.
- Jelassi et al., “Repeat After Me: Transformers are Better than State Space Models at Copying”, ICML 2024.