EXPLAINER

Prompt Caching

techniquefreshness: volatilesources not yet verified

Prompt caching lets a provider or serving engine reuse the computation for a repeated prompt prefix — such as a long system prompt or document — cutting latency and often cost on later requests.

Short answer: Put stable content first (instructions, tools, documents) and variable content last, so repeated prefixes hit the cache. Rules (minimum length, expiry, pricing, how to mark cacheable content) are provider-specific and change — check current documentation.

What is it?

If every request starts with the same long instructions or document, there's no need to process them from scratch each time. Prompt caching remembers the work for that repeated beginning, so follow-up requests are faster and, with many providers, cheaper.

Also known as: context caching, prefix caching, cache prompts, cached tokens, reduce LLM cost caching, cache_control, KV cache reuse, prompt cache hit.

Technical definition

Implementation reuses the KV Cache for identical token prefixes (self-hosted engines offer automatic prefix caching). Hosted APIs expose varying mechanisms: automatic or explicit cache markers, minimum prefix lengths, time-to-live windows and distinct pricing for cache writes and reads. Any change early in the prefix invalidates later cache hits. Cache effects on determinism, data retention and privacy should be reviewed in provider terms.

Entity type: technique · Freshness class: volatile.

When should I use it?

Valuable with long, repeated prefixes (RAG documents, agent tool lists, few-shot examples).

Implementation patterns

  • Order prompt: static instructions → tools → documents → dynamic user content.
  • Keep prefix byte-identical (no timestamps/IDs early).
  • Warm caches for high-traffic prompts.
  • Measure hit rate.

Technologies and frameworks

Commonly associated: provider caching features, vLLM automatic prefix caching. Names are examples of what exists, not recommendations, and projects change quickly.

Practical guide

  1. Read the provider docs.
  2. Reorder the prompt.
  3. Log cached-token counts.
  4. Tune for your traffic pattern.

Common failure modes

  • Dynamic data at the start breaking prefix matches.
  • Prefix below minimum length.
  • Cache expiry between sparse requests.
  • Assuming cached = free.

Troubleshooting

  • Low hit rate — Move variable text later; keep prefix identical; check minimum length and TTL.
  • Costs unexpectedly high — Compare cache-write and read pricing to your request frequency.

Contrasted with

  • KV Cache — The KV cache is the data structure; prompt caching is the feature of reusing it across requests.

Questions this page answers

  • What is prompt caching?
  • How do I reduce the cost of long system prompts?
  • Why is my prompt cache not hitting?
  • Is prompt caching safe for private data?

Typical goals: cut repeated-prompt cost and latency; structure prompts for cache hits.

Sources and verification

Status: not independently verified. This entry was authored on 2026-10-06 from general technical knowledge, without live checking of the sources below. Treat every source as a lead to confirm (identifier, title, authorship, current version) before publication or citation.

Freshness class: volatile — Products, versions, protocols, benchmarks, leaderboards, regulations and dates. Verify against the primary source before relying on specifics; re-review every 3 months.

Time-sensitive claims to verify before relying on them:

  • Pricing, minimum lengths, TTLs, markers and availability per provider.

Canonical sources (unverified leads):

  • Provider documentation on prompt/context caching (identify current pages) — locator to be identified
  • Kwon et al., PagedAttention (2023) for prefix sharing in serving — arXiv:2309.06180