Stanford CS25: Transformers United V6 I On the Tradeoffs of State Space Models and Transformers

Explosion of sub-quadratic alternatives

Since Mamba's release two years ago, the field has rapidly adopted linear-complexity architectures including Mamba 2/3, xLSTM, DeltaNet, and gated DeltaNet as production-viable alternatives to transformers.

Production-scale hybrid adoption

Major AI labs now deploy hybrid models (Jamba, Zamba, Samba, Qwen, Hunyuan, NeMo-Megatron) combining SSM layers with attention mechanisms, with several models scaled to hundreds of billions of parameters.

Convergent nomenclature

Terms like linear attention, modern RNNs, linear RNNs, and state space models now largely refer to the same family of input-dependent recurrent architectures with similar computational characteristics.

The KV cache bottleneck

Transformers function like expandable databases, maintaining a growing KV cache of every past token that enables precise pairwise comparisons but creates quadratic scaling in both memory and computation during inference.

Fixed-state compression paradigm

SSMs operate like brains, compressing all historical context into a fixed-size hidden state that remains constant regardless of sequence length, enabling linear time complexity and constant memory per generation step.

Architectural tradeoff fundamentals

The distinction between these approaches centers on what they store between generation steps: transformers cache raw tokens for exact lookup while SSMs maintain compressed summaries for efficient processing.

Expanded state dimensions

Modern SSMs expand input dimensions by 64-128x (state size of 64-128), creating a much wider information bottleneck than LSTMs to preserve critical information from dense modalities like language.

Input-dependent selectivity

Parameters become functions of the input itself (A and B matrices vary by token), allowing the model to dynamically control what information to remember or discard based on current context.

Parallel training algorithms

Efficient computation via associative scans (original Mamba) and chunked matrix multiplications (Mamba 2/DeltaNet) makes training these large-state models feasible despite their recurrent formulation.

Leading production variants

Mamba 2 and gated DeltaNet currently represent the most tried-and-true implementations, with gated DeltaNet offering greater modeling power at slightly reduced computational speed compared to Mamba 2.

Architectural convergence

Modern SSM variants share more structural similarities with each other than with attention mechanisms, differing primarily in specific parameterizations while maintaining the core linear-recurrent paradigm.

Framework for model selection

Choose Transformers for tasks requiring exact retrieval from long contexts and SSMs for efficient inference with compressed representations, with hybrid architectures offering practical middle-ground solutions.