EXPLAINER

FlashAttention

techniquefreshness: evolvingsources not yet verified

FlashAttention is an exact attention algorithm that reorganises computation to cut memory traffic, speeding up transformers and enabling longer sequences.

Short answer: FlashAttention computes exact attention in tiles that stay in fast on-chip memory, avoiding the huge intermediate attention matrix. It speeds training and inference and reduces memory without changing results (up to numerical differences).

What is it?

Standard attention writes a very large table of scores to slow memory. FlashAttention processes it in small blocks that fit in the chip's fast memory, so the same answer is computed faster and with less memory.

Also known as: flash attention, FlashAttention-2, FlashAttention-3, IO-aware attention, efficient attention, fused attention, memory efficient attention.

Technical definition

Dao et al. (2022) present an IO-aware exact attention using tiling and online softmax, reducing reads/writes between GPU HBM and SRAM; subsequent versions improve parallelism and hardware utilisation: FlashAttention-3 targets Hopper GPUs with asynchrony and FP8 (FlashAttention-3 (and FlashAttention-4)). For bounded-memory streaming with a rolling cache see Attention Sinks and StreamingLLM. It is exact attention (not an approximation), so quality is unaffected apart from floating-point differences. Typically enabled through framework or library flags rather than hand-written.

Entity type: technique · Freshness class: evolving.

When should I use it?

Use by default on supported GPUs for training and serving transformers.

Implementation patterns

  • Enable via framework attention backend.
  • Combine with mixed precision.
  • Use with GQA and paged KV in serving.

Technologies and frameworks

Commonly associated: flash-attn library, PyTorch scaled_dot_product_attention, Hugging Face attn_implementation, vLLM. Names are examples of what exists, not recommendations, and projects change quickly.

Practical guide

  1. Use the framework's native fused attention if available.
  2. Verify the backend in logs.
  3. Benchmark with realistic sequence lengths.

Common failure modes

  • Unsupported GPU/dtype combinations falling back silently.
  • Version mismatches between library, CUDA and PyTorch.
  • Expecting it to cut compute (it mainly cuts memory traffic).

Troubleshooting

  • Install/build errors — Match library, CUDA and PyTorch versions; prefer prebuilt wheels or the framework's built-in backend.
  • No speedup — Check hardware support and that the backend is actually selected.

Contrasted with

Questions this page answers

  • What is FlashAttention?
  • Is FlashAttention approximate?
  • How do I enable FlashAttention?
  • Why is attention memory-bound?

Typical goals: speed up training or inference; understand how longer contexts become feasible.

Sources and verification

Status: not independently verified. This entry was authored on 2026-10-06 from general technical knowledge, without live checking of the sources below. Treat every source as a lead to confirm (identifier, title, authorship, current version) before publication or citation.

Freshness class: evolving — Established techniques whose best practice shifts as research and tooling move. Re-review every 6 months.

Canonical sources (unverified leads):

  • Dao et al., FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness (2022) — arXiv:2205.14135
  • Dao, FlashAttention-2 (2023) — arXiv:2307.08691