Mamba: From Intuition to Proof
How delta-gated state-space models challenge the Transformer — from the intuition of selective memory to the parallel-scan proof.
Efficient Hybrid Models
Long-context AI is hitting a quadratic wall.
I research the architectures that break through it — state-space models, linear attention, hybrids, and latent-reasoning loops that could make long-document AI affordable for hospitals, labs, and researchers who don't own a GPU cluster.
Every new token, replayed vs. remembered
Both models read the same sentence. Self-attention re-reads every previous token, every step. A selective SSM asks one question per token — worth remembering? — writes “keys” into its fixed-size state, and lets “the” pass straight through.
Transformer · KV-cache grows with every token
Selective SSM · fixed state — keeps meaning, skips filler
14
architectures mapped in the Atlas
12
published deep-dives on Towards AI
1
survey paper in preparation
24 mo
empirical research plan
The problem
Attention pays for context twice — in compute and in memory.
Self-attention compares every token with every other token, so compute grows with the square of context length — and the key–value cache it keeps grows linearly on top of that. Drag the slider: the numbers below are computed live for a real model (Llama-3-8B, FP16) against a Mamba-2-class model whose recurrent state never grows at all.
—
token-pair comparisons (N²) per attention map
—
Transformer KV-cache (Llama-3-8B, FP16)
0.13 GB
SSM recurrent state — constant at any length
—
memory advantage at this context length
Inference memory vs. context length
KV-cache of Llama-3-8B vs. the fixed recurrent state of a Mamba-2-class 7B model. Hover for values; the marker tracks the slider.
Transformer KV-cache
SSM recurrent state
How these numbers are computed
KV-cache: 2 (K and V) × 32 layers × 8 KV heads × 128 head-dim × 2 bytes (FP16) = 128 KiB per token for Llama-3-8B with grouped-query attention. SSM state: ≈64 layers × 8192 inner-dim × 128 state-dim × 2 bytes ≈ 128 MiB, independent of context length. Attention scores: N² comparisons per layer-head pass.
And even with the compute…
Models lose the middle of long documents.
Even when a document fits in the context window, retrieval accuracy dips sharply when the relevant passage sits in the middle — the lost-in-the-middle failure documented by Liu et al. (2023). In clinical records, legal texts, and dense academic literature, the middle is where the evidence lives. My research plan measures exactly where this degradation begins, then tests whether selective state-space models can avoid it.
Retrieval accuracy vs. position of the relevant passage
Illustrative U-shaped curve after Liu et al. (2023), “Lost in the Middle: How Language Models Use Long Contexts.” Hover the points.
The pragmatic answer
Hybrids: keep a sliver of attention, make the rest linear.
Pure SSMs are cheap but fuzzy at verbatim recall; pure attention recalls perfectly but pays the quadratic tax on every layer. Production models now split the difference — a thin slice of real attention for precise, needle-in-haystack lookups, riding on a linear-state backbone that does the bulk of the work. Hover the stacks: orange layers are the only ones that keep a KV-cache.
≈8%
attention layers in Nemotron-H — the rest is Mamba-2 + FFN
6 / 52
attention layers in Nemotron 3 Nano 30B — 23 Mamba-2, 23 MoE
1 : 7
attention : Mamba ratio that Jamba shipped first
1M
token context window Nemotron 3 runs on this recipe
Where the attention actually is
Layer-by-layer composition, drawn to each model's real layer counts. The hybrid recipe: replace almost every attention layer with a constant-memory Mamba-2 layer, keep a few for high-fidelity recall.
Attention (keeps KV-cache)
Mamba-2 (constant state)
FFN (dense)
MoE (sparse experts)
Aug 2025 · Jet-Nemotron
“Gated DeltaNet achieves the best accuracy.”
NVIDIA's PostNAS ran six linear operators head-to-head — RWKV-7, RetNet, Mamba-2, GLA, DeltaNet, Gated DeltaNet — inside a frozen Transformer. Gated DeltaNet won; they built JetBlock on top of it.
My deep-dive: Inside Jet-Nemotron →
Dec 2025 · Nemotron 3
The frontier family ships on Mamba-2.
Four months later, the same lab bet its flagship open models — Nano, Super, and the 550B Ultra — on Mamba-2, not Gated DeltaNet, not JetBlock. What wins a 2B-scale search isn't what gets trusted at 550B.
My deep-dive: Nemotron 3's Mamba-MoE engine →Same lab. Same year. Two different winners. The field agrees on the hybrid recipe but not on the linear operator inside it — or where the attention layers belong, or whether any of it holds at the next order of magnitude. Nobody has a settled theory yet. That open question is precisely where my 24-month research plan and survey sit: put the operators under one notation, then measure — at kernel level — what the state actually remembers.
Update · Aug 2026 — Nemotron 3.5 Lightning. The bet hardened: NVIDIA's newest release is a 30B mixture-of-experts with just 3B active parameters, on the same interleaved Mamba-2 + MoE backbone with a few attention layers. The interesting part: its headline “up to 4× output speed” doesn't come from the backbone at all — it comes from decoding. Multi-token prediction was trained in (with a dedicated MTP-boosting phase), and two draft models — DSpark and DFlash — drive speculative decoding on top. The race below shows why that works. No technical report on arXiv yet; details from the NVIDIA announcement ↗. Multi-token prediction is a Phase-2 architecture in my Atlas — I wrote about it in Why Your LLM Should Be Guessing ↗.
Why Lightning is fast: guess four, check once
Both lanes decode the same sentence. Autoregression pays one full forward pass per token. Speculative decoding lets a tiny draft model propose four tokens, then the big hybrid verifies them all in a single pass — wrong guesses get thrown out and redrafted, and it still finishes ~3× sooner.
Verified one at a time — 1 token per pass
Drafted then verified — up to 4 tokens per pass
Two other levers
Spend fewer bits per weight, or fewer steps per answer.
Replacing attention with a fixed-size state is only one way to make long-context AI affordable. Two Atlas nodes attack the cost from completely different directions: 1-bit models shrink what a weight costs to store and multiply, and diffusion language models shrink how many forward passes an answer costs to produce. Both are in the survey; neither needs a bigger GPU.
Where these sit in the map: 1-bit / MatMul-free is a Phase-1 node, diffusion LLMs are Phase 3 — both are covered in the survey's coverage map.
Research proposal
Can state-space models make long-context reasoning affordable?
A 24-month empirical program: map the exact degradation curve of Transformer long-context retrieval, then test whether Mamba-2-class architectures can match it at a fraction of the memory — with hardware-level profiling on dedicated HPC infrastructure, and falsifiable success criteria fixed in advance.
≤ 5 pts
target retrieval-F1 gap vs. Transformer baseline beyond 32K tokens
≥ 40%
target reduction in peak KV-cache memory
3
phases: baseline · build · validate
O(N)
the complexity claim to be verified at kernel level
The Architecture Atlas
Fourteen architectures. Three phases. One story.
Every post-Transformer architecture answers a specific failure of the one before it. The Atlas maps that evolutionary chain from LSTM (1997) to recursive language models (2026) — each node with its motivation, key equation, complexity, and the paper behind it.
Phase 1
The Linear-Sequence Revolution
Replace the KV-cache with a fixed-size state — RNNs, 1-bit LLMs, linear attention, hybrids, and Mamba.
Phase 2
Latent Reasoning Loops
Spend adaptive test-time compute inside a latent state — TTT, multi-token prediction, HRM, TRM, recursive LMs.
Phase 3
Beyond Causal Autoregression
Abandon next-token prediction — energy-based Transformers, diffusion LLMs, JEPA, and world models.
Writing
Deep-dives, published in Towards AI.
Selected articles — the full list, mapped to Atlas nodes, is on the Writing page.
Gated Attention & DeltaNets
The missing link for long-context AI: fusing Mamba's forgetting with DeltaNet's precise memory writes.
Tiny Recursion Models
How a 7M-parameter network with recursion beats models a thousand times its size on hard reasoning puzzles.
In preparation
Beyond the KV-Cache: A Survey of Transformer-Alternative Language Model Architectures
A survey organizing the fourteen Atlas architectures into their three evolutionary phases, under a single unified state-update notation — in preparation for arXiv (cs.CL / cs.LG). The Atlas is its living companion — and every one of the fourteen gets its own published deep-dive first.
7 of 14 architectures already have a published deep-dive
See the coverage map →