EXPLAINER

Backpropagation and Gradient Descent

techniquefreshness: stablesources not yet verified

Gradient descent adjusts a model's parameters in the direction that reduces loss; backpropagation is the efficient way to compute those gradients through a neural network.

Short answer: Training = repeat: compute loss, backpropagate gradients, nudge parameters downhill. Learning rate, batch size and optimiser choice (SGD, Adam/AdamW) control speed and stability.

What is it?

To learn, a network measures how wrong it is (the loss), works out how each weight contributed to that error (backpropagation) and nudges each weight slightly to reduce it (gradient descent). Repeat millions of times.

Also known as: backprop, gradient descent, SGD, stochastic gradient descent, Adam optimizer, AdamW, learning rate, loss function.

Technical definition

Backpropagation applies the chain rule to compute ∂L/∂θ for all parameters in one backward pass. Optimisers: SGD with momentum, RMSProp, Adam (Kingma & Ba, 2014), AdamW (decoupled weight decay). Practical elements: learning-rate schedules and warm-up, gradient clipping, mini-batching, normalisation layers (batch/layer norm), mixed precision. Pathologies: vanishing/exploding gradients, saddle points, poor conditioning.

Entity type: technique · Freshness class: stable.

When should I use it?

Needed knowledge for anyone training or fine-tuning neural networks.

Implementation patterns

  • Learning-rate warm-up + cosine/linear decay.
  • Gradient clipping for stability.
  • Overfit one batch as a sanity test.
  • Log training and validation loss together.

Technologies and frameworks

Commonly associated: PyTorch autograd, JAX grad, optimizers: Adam, AdamW, SGD. Names are examples of what exists, not recommendations, and projects change quickly.

Practical guide

  1. Run the one-batch overfit test.
  2. Sweep learning rate on a log scale.
  3. Add schedule and clipping.
  4. Track gradient norms.
  5. Scale up only after small runs behave.

Common failure modes

  • Learning rate too high (diverges) or too low (stalls).
  • Forgetting to zero gradients or switch eval/train mode.
  • Exploding gradients in deep or recurrent nets.
  • Data/label bugs mistaken for optimisation problems.

Troubleshooting

  • Loss is NaN or spikes — Lower learning rate, clip gradients, check inputs for NaN/inf.
  • Loss flat from the start — Check labels, learning rate, initialisation and that parameters are in the optimiser.

Contrasted with

  • Reinforcement Learning — RL gets gradient estimates from reward feedback (e.g. policy gradients), not directly from labelled targets.

Questions this page answers

  • What is backpropagation?
  • What is gradient descent?
  • How do I choose a learning rate?
  • Why is my loss not decreasing?
  • What is the difference between SGD and Adam?

Typical goals: understand how neural networks are trained; tune learning rate and optimiser; debug training that will not converge.

Sources and verification

Status: not independently verified. This entry was authored on 2026-10-06 from general technical knowledge, without live checking of the sources below. Treat every source as a lead to confirm (identifier, title, authorship, current version) before publication or citation.

Freshness class: stable — Core ideas that change slowly (maths, classic algorithms). Re-review yearly.

Canonical sources (unverified leads):

  • Rumelhart, Hinton & Williams, Learning representations by back-propagating errors, Nature (1986) — doi:10.1038/323533a0
  • Kingma & Ba, Adam: A Method for Stochastic Optimization (2014) — arXiv:1412.6980
  • Loshchilov & Hutter, Decoupled Weight Decay Regularization (2017) — arXiv:1711.05101
Learning path: Machine learning foundations (step 3 of 9)
← Neural NetworksOverfitting and Regularization →