EXPLAINER

Speculative Decoding

techniquefreshness: evolvingsources not yet verified

Speculative decoding speeds up generation by letting a small draft model propose several tokens that a large model verifies in one pass, without changing the output distribution.

Short answer: A cheap draft model proposes k tokens; the big model checks them in parallel and accepts the matching prefix. Done correctly, output has the same distribution as the big model alone but arrives faster.

What is it?

Generating one word at a time is slow. A small helper model guesses the next few words; the big model then checks all of them at once and keeps the correct ones. When the helper guesses well, you get several words for the price of one big-model step.

Also known as: speculative sampling, draft model, assisted generation, Medusa, prompt lookup decoding, faster LLM inference, lookahead decoding.

Technical definition

Leviathan et al. (2022) and Chen et al. (2023) describe accept/reject sampling schemes that preserve the target model's distribution. Speedup depends on the draft's acceptance rate and relative cost. Variants include self-speculation (draft heads such as Medusa), n-gram/prompt-lookup drafts, and tree-based verification. Gains are workload-dependent and are largest on predictable text such as code or structured output.

Entity type: technique · Freshness class: evolving.

When should I use it?

Use for latency-sensitive serving at low-to-moderate batch sizes.

Implementation patterns

  • Pair a small and large model from the same family.
  • Use n-gram drafts for structured or repetitive output.
  • Tune draft length by measured acceptance rate.

Technologies and frameworks

Commonly associated: vLLM, llama.cpp, Hugging Face assisted generation, TensorRT-LLM. Names are examples of what exists, not recommendations, and projects change quickly.

Practical guide

  1. Check your engine supports it.
  2. Choose a draft from the same family.
  3. Benchmark with realistic prompts and batch sizes.
  4. Keep it only if latency improves.

Common failure modes

  • Low acceptance rate from a poorly matched draft.
  • Memory overhead of a second model.
  • Little gain at high batch sizes where the server is compute-bound.

Troubleshooting

  • No speedup — Measure acceptance rate; try a better draft or shorter speculation length.
  • Higher memory use — Use self-speculative or n-gram methods.

Contrasted with

  • Quantization of AI Models — Quantisation makes each step cheaper; speculative decoding reduces the number of expensive steps.

Questions this page answers

  • What is speculative decoding?
  • How does a draft model speed up LLMs?
  • Does speculative decoding change the output?
  • When does speculative decoding not help?

Typical goals: reduce LLM latency without changing outputs; check whether an inference engine feature will help my workload.

Sources and verification

Status: not independently verified. This entry was authored on 2026-10-06 from general technical knowledge, without live checking of the sources below. Treat every source as a lead to confirm (identifier, title, authorship, current version) before publication or citation.

Freshness class: evolving — Established techniques whose best practice shifts as research and tooling move. Re-review every 6 months.

Canonical sources (unverified leads):

  • Leviathan et al., Fast Inference from Transformers via Speculative Decoding (2022) — arXiv:2211.17192
  • Chen et al., Accelerating LLM Decoding with Speculative Sampling (2023) — arXiv:2302.01318