What is it?
Instead of teaching a model from zero, borrow one that already learned general skills — recognising shapes, understanding language — and teach it your specific task. It needs far less data and time.
Also known as: pretrained models, pre-trained model fine-tuning, feature extraction, domain adaptation, ImageNet pretrained, catastrophic forgetting, warm start.
Technical definition
Approaches: use the pre-trained model as a frozen feature extractor with a new head; fine-tune all or some layers; parameter-efficient methods (LoRA and Parameter-Efficient Fine-Tuning (PEFT)); continued pre-training on domain data. Yosinski et al. (2014) studied transferability of features by layer. Negative transfer can occur when source and target are very different; catastrophic forgetting can erase general abilities if fine-tuning is aggressive.
Entity type: technique · Freshness class: stable.
When should I use it?
Default approach for images, text and audio when data is limited.
Implementation patterns
- Freeze backbone, train head first; then unfreeze with a small learning rate.
- Use domain-specific pre-trained models when available.
- LoRA for large models.
- Evaluate for forgetting on a general benchmark.
Technologies and frameworks
Commonly associated: Hugging Face Hub, PyTorch / torchvision, timm. Names are examples of what exists, not recommendations, and projects change quickly.
Practical guide
- Pick a pre-trained model close to your domain.
- Check licence.
- Replicate its preprocessing.
- Train the head, then unfreeze gradually.
- Compare with a from-scratch baseline.
Common failure modes
- Source domain too different from target.
- Fine-tuning learning rate too high.
- Preprocessing mismatch with the pre-trained model.
- Ignoring licence terms of the pre-trained model.
Troubleshooting
- Accuracy collapses after fine-tuning — Lower learning rate, freeze early layers, mix in general data.
- Poor results immediately — Match tokenizer/normalisation/preprocessing to the pre-trained model.
Contrasted with
- What is Fine-Tuning an AI Model? — Fine-tuning is the most common form of transfer learning for language models.
Questions this page answers
- What is transfer learning?
- Should I freeze layers when fine-tuning?
- What is catastrophic forgetting?
- How much data do I need to fine-tune a pre-trained model?
Typical goals: adapt a pre-trained model to my data; decide between freezing and fine-tuning.
Sources and verification
Status: not independently verified. This entry was authored on 2026-10-06 from general technical knowledge, without live checking of the sources below. Treat every source as a lead to confirm (identifier, title, authorship, current version) before publication or citation.
Freshness class: stable — Core ideas that change slowly (maths, classic algorithms). Re-review yearly.
Canonical sources (unverified leads):
- Yosinski et al., How transferable are features in deep neural networks? (2014) — arXiv:1411.1792
- Pan & Yang, A Survey on Transfer Learning, IEEE TKDE (2010) — locator to be identified