Attention vs. Memory: Why Transformers Killed the RNN
The math behind the coup — vanishing gradients, the sequential bottleneck, and why attention treats a sequence as a fully connected graph.
Writing
Deep-dives on the post-Transformer landscape.
I publish these deep-dives in Towards AI on Medium. The architecture series below maps one-to-one onto the Atlas, and each version hosted here links back to the original publication.
The architecture series
In reading order — each piece picks up where the previous architecture failed.
Mamba: From Intuition to Proof
How delta-gated state-space models challenge the Transformer — selective memory, zero-order hold, and the parallel-scan trick that makes it trainable.
Gated Attention & DeltaNets: The Missing Link for Long-Context AI
Two papers land on the same fix: a gate that lets the model multiply noise by zero — and a delta rule that writes memory surgically instead of blindly.
Attention, But Smarter: Inside Jet-Nemotron's Hybrid Design
NVIDIA's PostNAS recipe: keep the few attention layers that matter, replace the rest — full-attention accuracy at a fraction of the cost.
Inside the Mamba-MoE Engine of Nemotron 3
The production proof: a hybrid Mamba–Transformer mixture-of-experts with a million-token context window, and what its design choices reveal.
Why Your LLM Should Be Guessing: Breaking the Sequential Curse ↗
Why one-token-at-a-time decoding is a straitjacket, and how multi-token prediction teaches models to plan ahead.
Hierarchical Reasoning Models: When 27M Parameters Outperform Chain-of-Thought Giants
A slow planner and a fast worker, recursing at different frequencies — and beating far larger models on ARC-AGI with ~1000 training examples.
Tiny Recursion Models (TRM): How Tiny Networks With Recursion Beat Large Models on Hard Puzzles
A single 2-layer network recursing on a latent scratchpad — 7M parameters, state-of-the-art generalization on Sudoku-Extreme, Maze-Hard, and ARC-AGI.
Foundations
Earlier pieces on the building blocks — read on Medium.
The Secret Sauce of Context Windows: Unpacking RoPE ↗
Rotary positional encoding, and why it decides how far a context window can stretch.
Why Residual Connections? ↗
From static skips to dynamic routing — rewiring deep networks with hyper-connections.
SigLIP vs. CLIP: Overcoming the Softmax Bottleneck ↗
Why a sigmoid loss beats softmax contrastive training at scale.
CLIP: Contrastive Language–Image Pre-training ↗
How contrastive pre-training unlocked zero-shot vision.
Everything in one place: kyouma45.medium.com ↗