In preparation · arXiv cs.CL / cs.LG

Beyond the KV-Cache: A Survey of Transformer-Alternative Language Model Architectures

Fourteen architectures, three evolutionary phases, one unified notation. The Architecture Atlas on this site is the survey's living companion — the paper gives each node the full mathematical treatment.

14 architectures 3 evolutionary phases 1 unified notation View coverage ↓

Abstract

Working abstract

Modeling long sequences with Transformers incurs a quadratic-time attention cost and a linearly growing key–value cache, which together bottleneck both training and inference. This has spurred a wave of architectures that abandon or augment the canonical attention-plus-autoregression recipe. This survey organizes fourteen representative architectures into three evolutionary phases: (1) the linear-sequence revolution, which replaces the KV-cache with a fixed-size recurrent state (RNNs, 1-bit and MatMul-free models, linear-attention variants, hybrids, and pure state-space models); (2) active in-context execution and latent reasoning loops, which spend adaptive test-time compute inside a latent state (test-time-training layers, multi-token prediction, hierarchical and tiny recursive reasoning models, recursive language models); and (3) paradigm shifts beyond causal autoregression (energy-based models, diffusion language models, joint-embedding predictive architectures, and world models). A unified notation expresses every method as a transformation over a hidden state, a set of latent variables, and/or an energy function, with a master comparison across training parallelism, inference complexity, state size, recall ability, and generation paradigm.

Core notation

The unifying idea

Nearly every Phase-1 model is a special case of a single gated matrix-state recurrence:

$$ S_t = G_t \odot S_{t-1} + k_t^{\top} v_t, \qquad o_t = q_t S_t $$

Linear attention drops the gate; GLA makes it data-dependent and diagonal; Mamba-2 restricts it to scalar-times-identity (the State-Space Duality); Gated DeltaNet replaces the additive write with a delta-rule correction. And the delta rule itself is provably a step of test-time gradient descent — which is exactly where Phase 2 begins. The survey's contribution is making that cross-walk explicit, in one notation, with the receipts.

The gate $G_t$ is the whole argument in one control: push it toward the identity and the state is preserved, push it toward zero and the state is overwritten. Every Phase-1 architecture in the survey is a different answer to who decides that value.

Figures in progress

One mechanism per phase, drawn.

Interactive drafts of the mechanism figures the survey argues from. Each one is the load-bearing claim of its phase — what the state does, what the loop does, and what happens when you drop autoregression or drop precision entirely.

Phase 1 · the state replaces the cache

The recurrence is associative, so training stays parallel even though inference is sequential — this is the property that makes a fixed-size state competitive at all. Deep-dive: Mamba: From Intuition to Proof →.

Phase 1 · how far precision can fall

The 1-bit / MatMul-free node: ternary weights collapse matrix multiplication into additions, and the accuracy gap at the 2B scale is about a point.

Phase 2 · depth of iteration instead of depth of parameters

Latent reasoning loops spend test-time compute inside the state rather than in emitted tokens. Deep-dive: Tiny Recursion Models →.

Phase 3 · generation without left-to-right

Diffusion language models drop causal autoregression altogether — the answer is denoised in parallel, so decoding cost is set by the number of denoising steps, not the number of tokens.

Coverage map

Fourteen architectures. One deep-dive each.

Every architecture in the survey gets its own published article first — the deep-dives are how each section earns its place. Seven are live on the Writing page; the rest are queued in survey order.

Phase 1 · The Linear-Sequence Revolution

Phase 2 · Latent Reasoning Loops

Phase 3 · Beyond Causal Autoregression

  • Energy-Based Transformers 2025
    article planned
  • Diffusion LLMs 2025–2026
    article planned
  • JEPA 2023–2025
    article planned
  • World Models 2018 →
    article planned

Project state

Status

  • Scope locked: the fourteen anchor architectures mapped in the Atlas, field state verified as of August 2026.
  • Skeleton complete: three-phase taxonomy, unified notation table, master comparison table, per-architecture template.
  • Drafting now: Phase-1 subsections, beginning from the published deep-dives on the Writing page.

Interested in the draft, or in collaborating? Reach me via Medium or GitHub.