What is it?
Attention compares every word with every other word, which gets expensive for very long inputs. State space models instead keep a running summary of what they've seen, so cost grows linearly with length. They can be much cheaper on long sequences but may be weaker at looking up exact earlier details.
Also known as: SSM, Mamba, S4, Mamba-2, selective state space, linear attention alternative, linear time sequence models, structured state space.
Technical definition
Structured SSMs (S4; Gu et al., 2021) model sequences with linear dynamical systems that can be computed as convolutions for training and recurrences for inference. Mamba (Gu & Dao, 2023) adds input-dependent (selective) parameters with a hardware-aware scan. Relative to attention, SSMs have constant-size state at inference (no growing KV Cache) but a bounded memory; hybrid models interleave attention and SSM layers. Empirical comparisons are model-, scale- and task-dependent.
Entity type: concept · Freshness class: volatile.
When should I use it?
Consider for very long sequences or constrained inference memory; default to transformers where ecosystem support matters.
Implementation patterns
- Hybrid stacks mixing attention and SSM layers.
- Use for long-sequence modalities (audio, genomics, long text).
- Evaluate on retrieval-heavy tasks specifically.
Technologies and frameworks
Commonly associated: Mamba reference implementation, Hugging Face Transformers, hybrid attention–SSM models. Names are examples of what exists, not recommendations, and projects change quickly.
Practical guide
- Define your sequence length and latency needs.
- Check available pre-trained models and serving support.
- Benchmark on your tasks, especially recall-heavy ones.
- Compare cost per token against a transformer.
Common failure modes
- Weaker exact recall of distant details.
- Immature tooling relative to transformers.
- Reported gains not replicating at your scale.
Troubleshooting
- Poor in-context retrieval — Consider hybrid architectures or add retrieval (What is RAG (Retrieval-Augmented Generation)?).
- Tooling gaps — Check serving-engine support before committing.
Contrasted with
- What is a Transformer? (Attention Explained) — Attention keeps all tokens accessible at quadratic cost; SSMs compress history into a state at linear cost.
- Recurrent Neural Networks and LSTMs — Modern SSMs can be trained in parallel unlike classic RNNs.
Questions this page answers
- What is Mamba?
- What is a state space model?
- Are state space models better than transformers?
- Why do SSMs scale linearly with sequence length?
Typical goals: understand transformer alternatives; evaluate SSMs for long context.
Sources and verification
Status: not independently verified. This entry was authored on 2026-10-06 from general technical knowledge, without live checking of the sources below. Treat every source as a lead to confirm (identifier, title, authorship, current version) before publication or citation.
Freshness class: volatile — Products, versions, protocols, benchmarks, leaderboards, regulations and dates. Verify against the primary source before relying on specifics; re-review every 3 months.
Time-sensitive claims to verify before relying on them:
- Which production models use SSM/hybrid designs.
- Comparative benchmark claims against transformers.
- Tooling and serving support status.
Canonical sources (unverified leads):
- Gu et al., Efficiently Modeling Long Sequences with Structured State Spaces (S4) (2021) — arXiv:2111.00396
- Gu & Dao, Mamba: Linear-Time Sequence Modeling with Selective State Spaces (2023) — arXiv:2312.00752
- Dao & Gu, Transformers are SSMs (Mamba-2) (2024) — arXiv:2405.21060