What is it?
Transformers can look back at any earlier word directly, which makes them precise but costly for very long inputs. State-space models carry a compact summary forward, which is cheaper but can lose details. Mixed designs combine them.
Also known as: transformer vs mamba, attention vs SSM, mamba vs transformer, transformer alternatives, are transformers going away.
Technical definition
| Transformer | State space model | |
|---|---|---|
| History access | Direct (attention over all tokens) | Compressed state |
| Training cost vs length | Quadratic (mitigated by kernels) | Near-linear |
| Inference memory | Grows with context (KV Cache) | Constant per layer |
| Exact recall / copying | Strong | Weaker (task-dependent) |
| Ecosystem and tooling | Mature | Younger |
| Typical use | General LLMs | Long sequences, efficiency-focused or hybrid models |
Claims of superiority are scale- and benchmark-dependent.
Entity type: concept · Freshness class: volatile.
When should I use it?
Use transformers by default; evaluate SSMs for long-sequence efficiency.
Implementation patterns
- Default to transformers for ecosystem; test SSM/hybrids when sequence length or memory dominate.
- Evaluate retrieval/copy tasks explicitly.
Technologies and frameworks
Commonly associated: Mamba, FlashAttention, hybrid models. Names are examples of what exists, not recommendations, and projects change quickly.
Practical guide
- Define sequence lengths.
- List candidate models.
- Benchmark quality + cost.
- Verify serving support.
Common failure modes
- Choosing on headline throughput without checking quality on your tasks.
- Ignoring tooling support.
Troubleshooting
- Unsure — Run both on your longest realistic inputs and compare quality, latency, memory.
Contrasted with
- What is a Transformer? (Attention Explained) — The dominant architecture.
- State Space Models (Mamba and S4) — The linear-time alternative.
Questions this page answers
- Mamba vs transformer: which is better?
- Will state space models replace transformers?
- Which architecture is better for long documents?
- What are hybrid attention-SSM models?
Typical goals: compare architectures for long context; decide whether to adopt SSM models.
Sources and verification
Status: not independently verified. This entry was authored on 2026-10-06 from general technical knowledge, without live checking of the sources below. Treat every source as a lead to confirm (identifier, title, authorship, current version) before publication or citation.
Freshness class: volatile — Products, versions, protocols, benchmarks, leaderboards, regulations and dates. Verify against the primary source before relying on specifics; re-review every 3 months.
Time-sensitive claims to verify before relying on them:
- Current state of model availability and benchmark comparisons.
Canonical sources (unverified leads):
- Gu & Dao, Mamba (2023) — arXiv:2312.00752
- Vaswani et al., Attention Is All You Need (2017) — arXiv:1706.03762