What is it?
A student who memorises past exam answers can ace those exams and fail new ones. A model that memorises its training data does the same. Regularisation is the set of tricks that discourage memorising and encourage learning general patterns.
Also known as: overfitting, underfitting, regularisation, regularization, dropout, weight decay, early stopping, bias variance tradeoff.
Technical definition
Overfitting reflects variance: low training error, high generalisation error. Diagnose with learning curves and validation metrics. Mitigations: more data, data augmentation, L2/weight decay, dropout (Srivastava et al., 2014), early stopping, model capacity reduction, ensembling, label smoothing, cross-validation for model selection. The opposite failure, underfitting, shows high error on both sets. Note that very large models can generalise well despite interpolating training data, so capacity alone does not determine overfitting.
Entity type: concept · Freshness class: stable.
When should I use it?
Always relevant when training; central to small-data work.
Implementation patterns
- Learning curves: plot train vs validation metric over data size and epochs.
- Early stopping on validation loss.
- Augmentation for images/audio.
- Cross-validation for model selection on small data.
Technologies and frameworks
Commonly associated: scikit-learn, PyTorch dropout, weight decay in AdamW, Albumentations (augmentation). Names are examples of what exists, not recommendations, and projects change quickly.
Practical guide
- Hold out validation and test sets properly.
- Plot learning curves.
- Add one regularisation measure at a time.
- Use the test set once at the end.
- Re-check after deployment (Model Drift and Production Monitoring).
Common failure modes
- Tuning on the test set, overfitting to it indirectly.
- Leakage that makes validation look good.
- Too much regularisation causing underfitting.
- Ignoring distribution shift between validation and real use.
Troubleshooting
- Validation loss rising while training loss falls — Stop earlier, add regularisation or data.
- Both losses high — Underfitting: increase capacity, train longer, improve features.
Contrasted with
- Transfer Learning — Starting from pre-trained weights is itself a strong regulariser for small datasets.
Questions this page answers
- What is overfitting?
- How do I prevent overfitting?
- What is dropout?
- What is the bias-variance tradeoff?
- Why does my model perform well on training but not test data?
Typical goals: diagnose poor generalisation; choose regularisation techniques.
Sources and verification
Status: not independently verified. This entry was authored on 2026-10-06 from general technical knowledge, without live checking of the sources below. Treat every source as a lead to confirm (identifier, title, authorship, current version) before publication or citation.
Freshness class: stable — Core ideas that change slowly (maths, classic algorithms). Re-review yearly.
Canonical sources (unverified leads):
- Srivastava et al., Dropout: A Simple Way to Prevent Neural Networks from Overfitting, JMLR (2014) — locator to be identified
- Hastie et al., The Elements of Statistical Learning, ch. 7 — hastie.su.domains/ElemStatLearn