EXPLAINER

Knowledge Distillation

techniquefreshness: volatilesources not yet verified

Distillation trains a smaller student model to imitate a larger teacher, capturing much of its behaviour at lower cost.

Short answer: Distillation = train a small student on a big teacher's outputs (soft probabilities or generated data). It can give cheaper, faster models for narrow tasks; check teacher licence terms and expect a quality gap on hard cases.

What is it?

A large model is the teacher; a smaller model is the student that learns to copy the teacher's answers. The student ends up much cheaper to run while keeping much of the teacher's skill on the tasks it was taught.

Also known as: model distillation, teacher student, distilled model, DistilBERT, distill a large model, synthetic data from a bigger model.

Technical definition

Classic distillation (Hinton et al., 2015) minimises divergence between student and teacher output distributions, often with temperature softening; DistilBERT (Sanh et al., 2019) is a well-known example. For LLMs, "distillation" often means supervised training on teacher-generated text or reasoning traces. Provider terms of service may restrict using model outputs to train competing models — verify before doing so.

Entity type: technique · Freshness class: volatile.

When should I use it?

Use for high-volume narrow tasks where cost or latency matters.

Implementation patterns

  • Generate task-specific data with a teacher, filter, fine-tune a small model.
  • Logit distillation where teacher logits are available (open models).
  • Evaluate student vs teacher on your task.

Technologies and frameworks

Commonly associated: TRL, DistilBERT, synthetic data pipelines. Names are examples of what exists, not recommendations, and projects change quickly.

Practical guide

  1. Check the teacher's licence/terms.
  2. Define the narrow task.
  3. Generate and filter data.
  4. Train and evaluate the student.
  5. Route hard cases to the teacher.

Common failure modes

  • Student inherits teacher errors and biases.
  • Terms-of-use violations.
  • Distribution gap between synthetic and real inputs.
  • Over-claiming equivalence with the teacher.

Troubleshooting

  • Student weak on hard cases — Add more hard examples; keep the teacher for escalation.

Contrasted with

Questions this page answers

  • What is knowledge distillation?
  • How do I distill a large model into a small one?
  • Can I train on GPT outputs?
  • Distillation vs quantization?

Typical goals: make a cheaper model for a narrow task; check legality of training on model outputs.

Sources and verification

Status: not independently verified. This entry was authored on 2026-10-06 from general technical knowledge, without live checking of the sources below. Treat every source as a lead to confirm (identifier, title, authorship, current version) before publication or citation.

Freshness class: volatile — Products, versions, protocols, benchmarks, leaderboards, regulations and dates. Verify against the primary source before relying on specifics; re-review every 3 months.

Time-sensitive claims to verify before relying on them:

  • Provider terms about using outputs to train other models.

Canonical sources (unverified leads):

  • Hinton et al., Distilling the Knowledge in a Neural Network (2015) — arXiv:1503.02531
  • Sanh et al., DistilBERT (2019) — arXiv:1910.01108
  • Provider terms of use regarding training on outputs — locator to be identified