EXPLAINER

Quantization of AI Models

techniquefreshness: volatilesources not yet verified

Quantisation stores and computes a model's numbers at lower precision — for example 8-bit or 4-bit — to cut memory and often speed up inference, at some quality cost.

Short answer: Quantisation reduces numeric precision of weights (and sometimes activations or the KV cache). It lets bigger models fit on smaller hardware; 8-bit is usually safe, 4-bit needs checking, and quality loss varies by model, method and task.

What is it?

Model weights are stored as numbers. Rounding them to fewer bits shrinks the model — sometimes by 2–4× or more — so it fits on a smaller GPU or laptop. The price is a small loss of accuracy that depends on how aggressively you round and which method you use.

Also known as: quantisation, 4-bit quantization, 8-bit, GPTQ, AWQ, GGUF, int8, int4.

Technical definition

Methods: post-training quantisation (GPTQ, Frantar et al., 2022; AWQ, Lin et al., 2023; LLM.int8(), Dettmers et al., 2022), quantisation-aware training, and file formats such as GGUF used by llama.cpp with various k-quant schemes. Weight-only quantisation reduces memory and memory-bandwidth-bound decoding time; activation and KV-cache quantisation address other bottlenecks. Outlier activations are a known difficulty. Effects on quality, especially for reasoning and long context, must be measured per model.

Entity type: technique · Freshness class: volatile.

When should I use it?

Essential for local inference and cost-efficient serving.

Implementation patterns

  • Start with 8-bit; test 4-bit variants on your tasks.
  • Use pre-quantised checkpoints from trusted sources.
  • Quantise the KV cache for long contexts when supported.
  • Evaluate quality before and after on your own prompts.

Technologies and frameworks

Commonly associated: llama.cpp (GGUF), bitsandbytes, AutoGPTQ / AWQ tooling, vLLM, ONNX Runtime. Names are examples of what exists, not recommendations, and projects change quickly.

Practical guide

  1. Estimate memory: parameters × bytes per weight plus KV cache and overhead.
  2. Pick format for your runtime.
  3. Benchmark quality and speed.
  4. Verify provenance of downloaded weights (AI Privacy and Security: What to Know Before You Paste Data).

Common failure modes

  • Quality loss unnoticed because only perplexity was checked.
  • Tool-use or reasoning degrading more than chat.
  • Untrusted quantised files from unknown uploaders.
  • Format incompatible with the runtime.

Troubleshooting

  • Outputs degrade after quantising — Try a higher-bit variant or different method; evaluate on task-specific tests.
  • Model will not load — Check that format and runtime versions match (e.g. GGUF with llama.cpp).

Contrasted with

  • Knowledge Distillation — Distillation trains a smaller model; quantisation shrinks the numbers of an existing model.

Questions this page answers

  • What is model quantization?
  • What is GGUF?
  • Does quantization reduce quality?
  • What is the difference between GPTQ and AWQ?
  • How much VRAM do I need to run a 7B model?

Typical goals: fit a model in less memory; choose a quantisation level or format.

Sources and verification

Status: not independently verified. This entry was authored on 2026-10-06 from general technical knowledge, without live checking of the sources below. Treat every source as a lead to confirm (identifier, title, authorship, current version) before publication or citation.

Freshness class: volatile — Products, versions, protocols, benchmarks, leaderboards, regulations and dates. Verify against the primary source before relying on specifics; re-review every 3 months.

Time-sensitive claims to verify before relying on them:

  • Which formats/methods are best supported by which runtimes.
  • Quality-vs-bits claims for specific models.
  • Memory requirement figures for specific models.

Canonical sources (unverified leads):

  • Frantar et al., GPTQ (2022) — arXiv:2210.17323
  • Lin et al., AWQ (2023) — arXiv:2306.00978
  • Dettmers et al., LLM.int8() (2022) — arXiv:2208.07339
Learning path: Running and serving models (step 1 of 8)
What is Local AI? Running Models on Your Own Device →