EXPLAINER

GPUs and AI Accelerators

conceptfreshness: volatilesources not yet verified

AI workloads run on hardware that does massive parallel matrix maths — GPUs, TPUs and other accelerators. Memory size and bandwidth often matter more than raw compute for LLMs.

Short answer: For LLMs the usual bottlenecks are memory capacity (fit weights + KV cache) and memory bandwidth (decode speed). Ecosystem matters too: CUDA (NVIDIA), ROCm (AMD), TPUs (Google), Apple Silicon unified memory. Match hardware to model size and workload.

What is it?

AI maths is mostly giant multiplications, which GPUs do very well. When choosing hardware, the first question is usually "does the model fit in memory?" and the second "how fast can it read that memory?"

Also known as: GPU for AI, CUDA, TPU, HBM, VRAM, NPU, AI chips, which GPU for LLM.

Technical definition

Accelerators provide high-throughput matrix units with HBM or other high-bandwidth memory. Decode-phase LLM inference is typically memory-bandwidth-bound; training is compute- and interconnect-bound. Precision formats (FP32, BF16/FP16, FP8, INT8/INT4) trade accuracy for throughput and memory (Quantization of AI Models). Software stacks (CUDA, ROCm, XLA for TPUs, Metal) determine library support. Multi-device scaling requires fast interconnects and parallelism strategies (Distributed Training (Data, Tensor and Pipeline Parallelism)). Specific products, prices and specifications change frequently and must be verified from vendors.

Entity type: concept · Freshness class: volatile.

When should I use it?

Needed for any self-hosted or training workload.

Implementation patterns

  • Memory estimate = parameters × bytes per weight + KV cache + overhead.
  • Rent bursts in the cloud; own hardware for steady load.
  • Prefer widely supported stacks unless cost differences justify porting.

Technologies and frameworks

Commonly associated: NVIDIA CUDA, AMD ROCm, Google TPU / XLA, Apple Metal, cloud GPU instances. Names are examples of what exists, not recommendations, and projects change quickly.

Practical guide

  1. Define model size, precision and concurrency.
  2. Estimate memory.
  3. Check framework/driver support.
  4. Compare rent vs buy over expected utilisation.
  5. Benchmark before committing.

Common failure modes

  • Buying hardware without checking software support.
  • Ignoring KV cache memory.
  • Underestimating power/cooling.
  • Comparing FLOPs figures at different precisions.

Troubleshooting

  • Out of memory — Quantise, shorten context, shard across GPUs, or use a smaller model.
  • Slow despite a powerful GPU — Check batch size, bandwidth limits and CPU data loading.

Contrasted with

Questions this page answers

  • What GPU do I need to run an LLM?
  • How much VRAM do I need?
  • What is CUDA?
  • What is a TPU?
  • Why is memory bandwidth important for LLMs?

Typical goals: choose hardware for training or inference; estimate memory requirements.

Sources and verification

Status: not independently verified. This entry was authored on 2026-10-06 from general technical knowledge, without live checking of the sources below. Treat every source as a lead to confirm (identifier, title, authorship, current version) before publication or citation.

Freshness class: volatile — Products, versions, protocols, benchmarks, leaderboards, regulations and dates. Verify against the primary source before relying on specifics; re-review every 3 months.

Time-sensitive claims to verify before relying on them:

  • Current hardware generations, specifications, pricing and availability.

Canonical sources (unverified leads):

  • Vendor documentation for CUDA, ROCm, TPU and Metal — locator to be identified
  • Jouppi et al., In-Datacenter Performance Analysis of a Tensor Processing Unit (2017) — arXiv:1704.04760
Learning path: Running and serving models (step 7 of 8)
← Model Serving and Inference InfrastructureLLM Cost Optimization →