EXPLAINER

FlashAttention-3 (and FlashAttention-4)

techniquefreshness: volatilekey claims checked 2026-10-09

FlashAttention-3 is an exact attention implementation redesigned for NVIDIA Hopper GPUs. It overlaps data movement and computation, interleaves matrix multiplication with softmax, and uses FP8 low-precision, reaching up to about 740 TFLOPs/s in FP16 on H100.

Short answer: FlashAttention-3 (Shah et al., 2024) makes exact attention faster on Hopper (H100) GPUs using three techniques: asynchrony (overlap data movement and compute with warp specialisation), interleaving matmul and softmax, and FP8 with block quantisation and incoherent processing. Paper claims: 1.5–2.0× over FlashAttention-2, up to ~740 TFLOPs/s FP16 (75% utilisation), ~1.2 PFLOPs/s FP8. A FlashAttention-4 package for Hopper and Blackwell also exists.

What is it?

FlashAttention already avoided wasting memory bandwidth. The next problem was that new GPUs (the H100) can do several things at once, and FlashAttention-2 left much of that on the table. FlashAttention-3 reorganises the work so loading data, multiplying matrices and computing softmax overlap instead of waiting for each other, and it can use the GPU's faster 8-bit floating-point maths. The answer is the same attention, computed faster.

Also known as: FlashAttention-3, FlashAttention 3, FA3, flash attention 3, FlashAttention-4, FA4, Hopper attention kernel, H100 attention.

Technical definition

The paper (Shah, Bikshandi, Zhang, Thakkar, Ramani and Dao, 2024; arXiv:2407.08608) states that FlashAttention-2 achieved only 35% utilisation on H100 and proposes three techniques for Hopper GPUs: exploiting the asynchrony of Tensor Cores and the Tensor Memory Accelerator (TMA) to overlap computation and data movement through warp specialisation; interleaving block-wise matrix multiplication and softmax; and block quantisation with incoherent processing to exploit hardware FP8 support. Reported results: 1.5–2.0× speedup on H100 with FP16 reaching up to 740 TFLOPs/s (75% utilisation), FP8 close to 1.2 PFLOPs/s, and FP8 FlashAttention-3 with 2.6× lower numerical error than a baseline FP8 attention. These are the authors' measurements on specific hardware and shapes. The repository README (as read on the check date) describes FlashAttention-3 as a beta release for H100/H800 GPUs, CUDA 12.3 or later (12.8 recommended), supporting FP16/BF16 forward and backward and FP8 forward; and describes FlashAttention-4, written in CuTeDSL, as optimised for Hopper and Blackwell GPUs and installable as flash-attn-4. Because FlashAttention-3 is exact attention in the same sense as FlashAttention, quality is unchanged apart from low-precision effects when FP8 is used.

Entity type: technique · Freshness class: volatile.

When should I use it?

Relevant when training or serving transformers on H100/H800 (or newer) GPUs. Usually enabled through a framework rather than hand-written.

Implementation patterns

  • Let the framework or serving engine select the best attention backend for your GPU.
  • Benchmark FP16/BF16 first; trial FP8 only with an accuracy check.
  • Pin library, CUDA and PyTorch versions together.

Technologies and frameworks

Commonly associated: flash-attn library, Hopper (H100/H800), Tensor Cores, TMA, FP8, CuTeDSL, PyTorch SDPA. Names are examples of what exists, not recommendations, and projects change quickly.

Practical guide

  1. Identify your GPU architecture.
  2. Use your framework's default attention backend and read its logs for the kernel chosen.
  3. Benchmark with your real sequence lengths and batch sizes.
  4. Evaluate quality before any FP8 use.
  5. Re-check the README for current requirements and status.

Common failure modes

  • Expecting Hopper-only speedups on older GPUs.
  • Installing a beta build into production without testing.
  • Turning on FP8 without checking quality.
  • Quoting peak TFLOPs from the paper as expected end-to-end speedup.

Troubleshooting

  • Build or import errors — Match the library with CUDA (12.3+ per the README), PyTorch and GPU architecture.
  • No speedup — Check the GPU is Hopper and the backend is actually selected; other GPUs need other kernels.
  • Accuracy drop with FP8 — Fall back to BF16 or evaluate on your own tasks.

Contrasted with

  • FlashAttention — The original IO-aware algorithm; FlashAttention-3 targets Hopper-specific hardware features.
  • Quantization of AI Models — FP8 attention quantises the attention computation; weight quantisation shrinks stored weights.

Questions this page answers

  • What is FlashAttention-3?
  • How is FlashAttention-3 different from FlashAttention-2?
  • Does FlashAttention-3 work on A100 GPUs?
  • Is FP8 attention less accurate?
  • What is FlashAttention-4?

Typical goals: understand what changed from FlashAttention-2 to 3; decide whether FlashAttention-3 applies to your GPUs; use FP8 attention safely.

Sources and verification

Status: key claims checked on 2026-10-09 against the primary sources listed below; everything else is not independently verified. The claims under "Checked against primary sources" were compared with the linked pages on that date. Pages change, so re-check anything time-sensitive before relying on it.

Checked against primary sources (2026-10-09):

  • Abstract: FlashAttention-2 reached only 35% utilisation on H100; three techniques (asynchrony with warp specialisation, interleaved matmul/softmax, FP8 block quantisation with incoherent processing).
  • Reported results: 1.5–2.0× speedup, FP16 up to 740 TFLOPs/s (75% utilisation), FP8 close to 1.2 PFLOPs/s, 2.6× lower numerical error than a baseline FP8 attention.
  • Authors: Shah, Bikshandi, Zhang, Thakkar, Ramani, Dao; submitted July 2024.
  • Repository README: FlashAttention-3 beta for H100/H800, CUDA >= 12.3 (12.8 recommended), FP16/BF16 forward and backward, FP8 forward; FlashAttention-4 in CuTeDSL for Hopper and Blackwell, install flash-attn-4.

Primary sources consulted:

Not independently verified:

  • Independent reproductions of the speedups.
  • The README's beta status may have changed since it was read.
  • FlashAttention-4 has no paper cited here; its claims rest only on the README line.
  • Which serving engines enable FlashAttention-3 by default.

Freshness class: volatile — Products, versions, protocols, benchmarks, leaderboards, regulations and dates. Verify against the primary source before relying on specifics; re-review every 3 months.

Time-sensitive claims to verify before relying on them:

  • Beta/stable status and supported GPUs of FlashAttention-3.
  • FlashAttention-4 maturity and hardware support.
  • CUDA and PyTorch version requirements.
  • Benchmark numbers on newer GPUs.

Further reading and canonical sources (not all checked):

  • Shah et al., FlashAttention-3: Fast and Accurate Attention with Asynchrony and Low-precision (2024) — arXiv:2407.08608
  • flash-attention repository README — github.com/Dao-AILab/flash-attention
  • Dao et al., FlashAttention (2022) — arXiv:2205.14135