EXPLAINER

KV Cache

techniquefreshness: evolvingsources not yet verified

The KV cache stores attention keys and values for past tokens so a model need not recompute them while generating, trading memory for speed.

Short answer: During generation, the model caches keys and values of earlier tokens, so each new token only computes its own attention. The cache grows with context length and batch size, making memory the main serving limit.

What is it?

When a model writes a long answer, it would be wasteful to re-read the whole conversation for every new word. It stores intermediate results (the "KV cache") from earlier words and reuses them. That makes generation faster but uses a lot of memory as the conversation grows.

Also known as: key value cache, KV caching, attention cache, grouped query attention, GQA, multi-query attention, PagedAttention, prefix cache.

Technical definition

For each layer, keys and values of previous tokens are retained, giving per-token decode cost linear in context but memory O(layers × heads × head_dim × tokens × batch). Techniques to reduce or manage it: multi-query and grouped-query attention (Shazeer, 2019; Ainslie et al., 2023), PagedAttention (Kwon et al., 2023, vLLM), cache quantisation, prefix caching and eviction/compression policies. Prompt-caching features offered by providers reuse computed prefixes (Prompt Caching).

Entity type: technique · Freshness class: evolving.

When should I use it?

Matters when serving LLMs or when you hit memory limits at long context.

Implementation patterns

  • Paged KV memory management.
  • GQA/MQA models to shrink cache size.
  • Prefix/prompt caching for shared system prompts.
  • KV quantisation for long contexts.

Technologies and frameworks

Commonly associated: vLLM PagedAttention, llama.cpp, TensorRT-LLM. Names are examples of what exists, not recommendations, and projects change quickly.

Practical guide

  1. Estimate cache size per token for your model.
  2. Multiply by context and concurrency.
  3. Choose a serving engine with paged KV.
  4. Stabilise shared prefixes.

Common failure modes

  • Out-of-memory at long context or high concurrency.
  • Fragmentation wasting memory.
  • Cache invalidated by tiny prompt changes.

Troubleshooting

  • OOM with many concurrent users — Reduce max context/batch, use paged KV engines, quantise cache, or use a GQA model.
  • Latency high on repeated prompts — Use prefix caching and keep shared prefixes stable.

Contrasted with

  • Prompt Caching — Prompt caching is a provider billing/latency feature built on reusing cached prefixes; the KV cache is the underlying data structure.

Questions this page answers

  • What is a KV cache?
  • Why does GPU memory grow with context length?
  • What is grouped-query attention?
  • What is PagedAttention?

Typical goals: understand memory use during inference; optimise long-context serving.

Sources and verification

Status: not independently verified. This entry was authored on 2026-10-06 from general technical knowledge, without live checking of the sources below. Treat every source as a lead to confirm (identifier, title, authorship, current version) before publication or citation.

Freshness class: evolving — Established techniques whose best practice shifts as research and tooling move. Re-review every 6 months.

Canonical sources (unverified leads):

  • Kwon et al., Efficient Memory Management for LLM Serving with PagedAttention (2023) — arXiv:2309.06180
  • Ainslie et al., GQA: Training Generalized Multi-Query Transformer Models (2023) — arXiv:2305.13245
  • Shazeer, Fast Transformer Decoding: One Write-Head is All You Need (2019) — arXiv:1911.02150
Learning path: Running and serving models (step 5 of 8)
← vLLMModel Serving and Inference Infrastructure →