EXPLAINER

Model Serving and Inference Infrastructure

conceptfreshness: volatilesources not yet verified

Serving turns a trained model into a reliable service: loading it on suitable hardware, batching requests, managing memory, scaling, and meeting latency and cost targets.

Short answer: Key concerns: latency (time to first token, tokens/second), throughput, memory (weights + KV cache), batching, scaling and cost per request. Use a serving engine suited to your hardware and traffic, and load-test with realistic prompts.

What is it?

Running a model for real users is an engineering problem: it must answer quickly, handle many people at once and not cost a fortune. Serving software and GPU choices decide that.

Also known as: LLM serving, inference server, inference optimization, time to first token, TTFT, tokens per second, throughput vs latency, model deployment.

Technical definition

LLM inference has a compute-bound prefill phase and a memory-bandwidth-bound decode phase. Techniques: continuous batching, paged KV cache (KV Cache), quantisation (Quantization of AI Models), speculative decoding (Speculative Decoding), tensor/pipeline parallelism, prefix caching, request routing and autoscaling. Metrics: TTFT, inter-token latency, tokens/s, GPU utilisation and cost per million tokens. Engines include vLLM, llama.cpp-based servers, TensorRT-LLM, and managed endpoints from cloud and model providers; verify features per version.

Entity type: concept · Freshness class: volatile.

When should I use it?

Relevant to anyone self-hosting models or choosing managed endpoints.

Implementation patterns

  • Separate gateway (auth, rate limits, routing) from model servers.
  • Autoscale on queue depth or GPU utilisation.
  • Warm pools to avoid cold-start latency.
  • Canary new model versions.
  • Cache repeated prefixes/responses where safe.

Technologies and frameworks

Commonly associated: vLLM, TensorRT-LLM, Triton Inference Server, KServe, Ray Serve, managed model endpoints. Names are examples of what exists, not recommendations, and projects change quickly.

Practical guide

  1. Collect real prompt/response length distributions.
  2. Estimate memory: weights + KV cache × concurrency.
  3. Choose engine and hardware.
  4. Load-test and set SLOs.
  5. Add monitoring and autoscaling.

Common failure modes

  • Cold starts of large models.
  • OOM from long contexts and high concurrency.
  • Benchmarking with unrealistic prompts.
  • Single point of failure.
  • No back-pressure or timeouts.

Troubleshooting

  • High tail latency — Limit max context/output; tune batching; add replicas.
  • Cost too high — Quantise, route easy requests to smaller models, cache (LLM Cost Optimization).

Contrasted with

Questions this page answers

  • How do I serve an LLM in production?
  • What is time to first token?
  • How much GPU memory do I need to serve a model?
  • How do I scale LLM inference?

Typical goals: size infrastructure for an LLM; reduce latency and cost.

Sources and verification

Status: not independently verified. This entry was authored on 2026-10-06 from general technical knowledge, without live checking of the sources below. Treat every source as a lead to confirm (identifier, title, authorship, current version) before publication or citation.

Freshness class: volatile — Products, versions, protocols, benchmarks, leaderboards, regulations and dates. Verify against the primary source before relying on specifics; re-review every 3 months.

Time-sensitive claims to verify before relying on them:

  • Engine feature sets, hardware support and cost/performance claims.

Canonical sources (unverified leads):

  • Kwon et al., PagedAttention / vLLM (2023) — arXiv:2309.06180
  • Serving engine documentation (vLLM, Triton, TensorRT-LLM) — locator to be identified
Learning path: Running and serving models (step 6 of 8)
← KV CacheGPUs and AI Accelerators →