Research proposal
Developing compute-efficient alternatives to Transformer architectures
Mitigating context degradation and quadratic scaling via selective state-space models — a 24-month empirical investigation with falsifiable success criteria fixed in advance.
Two failures, one architecture
Every major LLM today rests on the Transformer, and the Transformer carries a structural flaw: self-attention compares every token with every other token, so compute scales as O(N²) in sequence length while the key–value cache grows without bound. Long-context reasoning becomes computationally prohibitive for everyone but the most resource-rich institutions — it gatekeeps advanced AI from hospitals, universities, and independent researchers.
Worse, even when the compute is available, Transformers exhibit a reliability failure at long contexts: the lost-in-the-middle problem, where models systematically ignore information buried in the center of large documents and produce confidently wrong outputs — precisely in the high-stakes domains (clinical records, legal texts, dense academic literature) where reliability is non-negotiable. Current LLMs are simultaneously too expensive to deploy widely and too unreliable to trust deeply.
Hypothesis — carefully scoped
Selective state-space models (SSMs), specifically Mamba-2-class architectures, compress history into a rolling hidden state via dynamic Δ-gating and parallel scan — linear time, constant memory, no KV-cache. I hypothesize that for structured reasoning over dense documents, SSMs can reach parity with Transformer baselines in retrieval accuracy while sharply reducing inference latency, memory, and total compute.
The plan
Twenty-four months, three phases
Quantify the degradation, fix the baselines
Evaluate state-of-the-art open Transformers (Llama-3 variants) across 4K, 16K, 32K, and 128K token windows on RULER, LongBench, and ∞BENCH, extended with domain-specific academic and clinical corpora. Map the exact curve where positional encoding and self-attention begin to fail, and establish strict baseline metrics: inference latency, peak KV-cache memory, total compute.
Deploy and optimize the SSM challenger
Mamba-2 models at the 7B scale — matched to the Llama-3-8B baseline so the comparison is at equivalent capacity. Tune the input-dependent selection mechanism (dynamic Δ-gating) for the retention/compute balance; test whether parallel-scan processing handles dense technical text without any attention blocks; compare against hybrids (Jamba, Griffin) to isolate the contribution of eliminating the quadratic bottleneck entirely, grounded in Mamba-2's State-Space Duality framework.
A/B validation at the hardware level
Rigorous A/B testing against the Phase-1 baselines, with kernel-level profiling — FLOP counts and KV-cache footprints measured at scale — to empirically verify the O(N) efficiency claim, not just cite it. Findings compiled into a peer-reviewed submission on SSMs as a practically viable low-compute alternative for long-context reasoning, with direct implications for edge deployment in clinical and assistive settings.
Success criteria — set before the experiments
Why dedicated HPC infrastructure matters
The claim at the heart of this proposal — O(N) scaling with constant memory — is a hardware claim, and validating it honestly requires kernel-level memory-bandwidth visibility that commercial cloud providers do not expose. Profiling Mamba's parallel-scan kernels at scale needs a dedicated HPC environment (TSUBAME-class supercomputing infrastructure) where the interconnect and memory hierarchy are fully observable. That requirement shapes where this research can credibly be done.
Why it matters
Low-compute long-context NLP changes who gets to use serious AI. Validated SSM architectures could run assistive document-analysis systems on edge hardware in clinics and residential care — urgent in super-aging societies — without the energy-intensive cloud infrastructure Transformers demand. And by shrinking the inference requirement from multi-GPU clusters to a single GPU, the same result lowers the cost barrier for universities and independent researchers everywhere advanced AI is currently out of reach.
The empirical basis for this proposal came from building Amadeus, my agentic research assistant for academic document analysis: at production scale, KV-cache growth and mid-context retrieval failures are not theoretical concerns — they are the two walls every long-document system hits first.