EXPLAINER

Attention Sinks and StreamingLLM

techniquefreshness: evolvingkey claims checked 2026-10-09

Attention sinks are the first few tokens of a sequence that transformers attend to heavily even when they carry little meaning. StreamingLLM keeps those tokens plus a recent window in the KV cache so a model can stream indefinitely without collapsing.

Short answer: Language models give a lot of attention to the first few tokens even when they are not semantically important — attention sinks. If a sliding-window KV cache evicts them, quality collapses. StreamingLLM keeps the sink tokens (4 sufficed in the paper) plus the most recent tokens, enabling stable streaming over very long inputs without fine-tuning. It does not make the model remember evicted text.

What is it?

When a chat or stream gets longer than the memory a model keeps, the usual trick is a sliding window: forget the oldest words, keep the newest. Oddly, that makes the model fall apart — because the very first words act as an anchor the attention mechanism leans on. The fix is simple: always keep those first few tokens, plus the recent ones. The model then stays stable for a very long stream, though it still cannot recall what was thrown away.

Also known as: attention sink, StreamingLLM, streaming LLM, sink tokens, sliding window attention failure, infinite context streaming, rolling KV cache, first token attention.

Technical definition

Xiao, Tian, Chen, Han and Lewis (ICLR 2024; arXiv:2309.17453) observe that window attention, which caches only the most recent keys and values, fails once text length exceeds the cache size, and that keeping the key/value states of the initial tokens largely recovers performance; they call this phenomenon an "attention sink". StreamingLLM keeps the sink tokens' KV (four initial tokens sufficed in their experiments) together with a rolling window of recent tokens, letting models trained with a finite window (Llama-2, MPT, Falcon, Pythia are named) run stable language modelling for up to 4 million tokens and more without fine-tuning, with up to 22.2× speedup over the sliding-window-with-recomputation baseline. They also report that pre-training with a dedicated placeholder "sink token" improves streaming deployment. Implications that follow from the design (not additional claims from the paper): evicted tokens are not available to later attention, so the method stabilises long streams but does not extend effective memory; and sink tokens work with the KV-cache management used in serving engines (KV Cache). Exact positional handling and implementation details vary between libraries.

Entity type: technique · Freshness class: evolving.

When should I use it?

Use when serving unbounded streams (long chats, logs) with a bounded memory budget and when losing old details is acceptable.

Implementation patterns

  • Keep the first N tokens (sinks) plus the last W tokens in the cache; evict the middle.
  • Pre-train or fine-tune with a dedicated sink token if you control training.
  • Pair with summaries or retrieval when old facts must stay reachable (What is RAG (Retrieval-Augmented Generation)?).

Technologies and frameworks

Commonly associated: StreamingLLM, rolling KV cache, sink token (pre-training), sliding window cache. Names are examples of what exists, not recommendations, and projects change quickly.

Practical guide

  1. Measure perplexity or task quality with a plain sliding window to confirm the failure.
  2. Switch to sink + recent-window caching and compare.
  3. Set the number of sink tokens (the paper defaults to four) and the window size to your memory budget.
  4. Test on tasks that need recall, not only fluency.

Common failure modes

  • Evicting the first tokens in a sliding window and seeing perplexity explode.
  • Assuming a model can answer questions about evicted text.
  • Reporting "4M tokens" as the usable context for tasks that need long-range recall.
  • Mixing sink handling with position schemes that the implementation does not support.

Troubleshooting

  • Output degrades after the cache fills — Keep the initial tokens in the cache rather than evicting oldest-first.
  • Model forgets earlier facts — Expected: use retrieval, summaries or a longer window.
  • Library flag has no effect — Check the model architecture and library version support the sink-aware cache.

Contrasted with

  • What is a Context Window? — A larger window increases how much the model can attend to; attention sinks keep a bounded cache stable.
  • KV Cache — The KV cache is the storage; attention sinks are a policy for which entries to keep.
  • State Space Models (Mamba and S4) — SSMs change the architecture to avoid growing caches; sinks keep standard attention workable with a bounded cache.

Questions this page answers

  • What is an attention sink?
  • What is StreamingLLM?
  • Why does sliding window attention break when the first tokens are dropped?
  • Does StreamingLLM give infinite context?
  • How many sink tokens does StreamingLLM keep?

Typical goals: keep a model stable on endless input streams; understand why dropping the first tokens breaks generation; reduce KV cache memory for long chats.

Sources and verification

Status: key claims checked on 2026-10-09 against the primary sources listed below; everything else is not independently verified. The claims under "Checked against primary sources" were compared with the linked pages on that date. Pages change, so re-check anything time-sensitive before relying on it.

Checked against primary sources (2026-10-09):

  • Window attention fails once text exceeds the cache size; keeping initial tokens' KV largely recovers performance (the "attention sink" observation).
  • StreamingLLM enables finite-window models to generalise to infinite lengths without fine-tuning; Llama-2, MPT, Falcon and Pythia named; up to 4 million tokens and more.
  • A dedicated placeholder sink token in pre-training can improve streaming; up to 22.2× speedup over sliding-window recomputation.
  • Four initial tokens are used as attention sinks by default in the experiments; accepted at ICLR 2024 per the arXiv comment.

Primary sources consulted:

Not independently verified:

  • The statement that evicted tokens are unavailable and that the method does not extend memory is an inference from the method, not a quotation (the repository FAQ was not read).
  • Whether specific serving libraries implement sink-aware caches today.
  • Why sinks form (explanations in later literature were not reviewed).

Freshness class: evolving — Established techniques whose best practice shifts as research and tooling move. Re-review every 6 months.

Further reading and canonical sources (not all checked):

  • Xiao et al., Efficient Streaming Language Models with Attention Sinks (ICLR 2024) — arXiv:2309.17453
  • StreamingLLM repository (MIT HAN Lab) — github.com/mit-han-lab/streaming-llm