EXPLAINER

Scaling Laws

conceptfreshness: evolvingsources not yet verified

Scaling laws describe how model loss tends to improve predictably with more parameters, data and compute — and guide how to split a training budget.

Short answer: Empirical power-law relationships between loss and model size, data and compute let labs plan training runs. They predict loss, not every capability, and recommended ratios (e.g. Chinchilla) have been revisited as practice changed.

What is it?

Researchers found that making models bigger, training them on more text and using more compute improves them in a surprisingly smooth, predictable way. Scaling laws are those relationships. They help decide how to spend a training budget — for example how big a model to train and on how much data.

Also known as: neural scaling laws, Chinchilla scaling, compute optimal training, Kaplan scaling laws, bigger models better, power law loss, tokens per parameter.

Technical definition

Kaplan et al. (2020) reported power-law scaling of cross-entropy loss with parameters, dataset size and compute. Hoffmann et al. (2022, "Chinchilla") argued that, for a fixed compute budget, parameters and training tokens should be scaled roughly in proportion, and that many earlier models were under-trained for their size. Scaling laws concern pre-training loss on a distribution; mapping to downstream capabilities is less reliable, and inference-time techniques add another axis (Test-Time Compute (Inference-Time Scaling)). Later work considers inference cost, data quality and repeated data, so "optimal" ratios depend on assumptions.

Entity type: concept · Freshness class: evolving.

When should I use it?

Relevant to people planning pre-training; for application builders it explains why model families come in size tiers.

Implementation patterns

  • Fit scaling curves from small runs to extrapolate large-run loss.
  • Choose model size and token count jointly for a compute budget.
  • Over-train smaller models when inference cost dominates.

Technologies and frameworks

Commonly associated: training-run planning, compute budget calculators. Names are examples of what exists, not recommendations, and projects change quickly.

Practical guide

  1. Run small-scale experiments across sizes.
  2. Fit loss vs compute.
  3. Account for data limits.
  4. Weigh inference cost.
  5. Validate on downstream evaluations.

Common failure modes

  • Extrapolating beyond the tested range.
  • Assuming loss improvements translate to every downstream skill.
  • Ignoring data quality and repetition.
  • Treating a published optimal ratio as universal.

Troubleshooting

  • Large run underperforms prediction — Check data quality, hyperparameters and instabilities; verify the fitted range.
  • Unsure what size to train — Include inference cost in the objective, not training compute alone.

Contrasted with

Questions this page answers

  • What are scaling laws in AI?
  • What is Chinchilla optimal?
  • Do scaling laws mean bigger is always better?
  • How many tokens should a model be trained on?

Typical goals: understand why bigger models improve; reason about training budgets.

Sources and verification

Status: not independently verified. This entry was authored on 2026-10-06 from general technical knowledge, without live checking of the sources below. Treat every source as a lead to confirm (identifier, title, authorship, current version) before publication or citation.

Freshness class: evolving — Established techniques whose best practice shifts as research and tooling move. Re-review every 6 months.

Canonical sources (unverified leads):

  • Kaplan et al., Scaling Laws for Neural Language Models (2020) — arXiv:2001.08361
  • Hoffmann et al., Training Compute-Optimal Large Language Models (2022) — arXiv:2203.15556