EXPLAINER

Self-Supervised Learning

techniquefreshness: stablesources not yet verified

Self-supervised learning creates its own training signal from raw data — predicting hidden words, masked patches or the next token — and is how most foundation models are pre-trained.

Short answer: The model learns by solving tasks made from the data itself (predict the next token, fill in a masked word or patch). No human labels are needed, which makes training on huge corpora feasible.

What is it?

Hide part of the data and ask the model to predict it: the next word in a sentence, a masked patch of an image. The data supplies its own answers, so you can learn from enormous unlabelled collections. This is how large language models are pre-trained.

Also known as: SSL, self supervised, pretext task, masked language modeling, masked language modelling, next token prediction pretraining, masked autoencoder.

Technical definition

Pretext objectives include autoregressive next-token prediction (GPT-style), masked language modelling (BERT), masked autoencoding (MAE for images), and contrastive objectives (Contrastive Learning and CLIP). The learned representations are then adapted by What is Fine-Tuning an AI Model? or used frozen. Quality depends on data scale, diversity and objective design.

Entity type: technique · Freshness class: stable.

When should I use it?

Relevant whenever you use or adapt foundation models; rarely something an application team trains from scratch.

Implementation patterns

  • Pre-train on large unlabelled data, then fine-tune on small labelled data.
  • Use frozen self-supervised embeddings with a small classifier head.
  • Continue pre-training on domain text before task fine-tuning.

Technologies and frameworks

Commonly associated: BERT-style masking, GPT-style autoregression, MAE, SimCLR. Names are examples of what exists, not recommendations, and projects change quickly.

Practical guide

  1. Choose a pre-trained model where possible instead of pre-training from scratch.
  2. Match the pretext objective family to your modality.
  3. Evaluate on your downstream task.
  4. Check pre-training data licensing and bias.

Common failure modes

  • Pre-training data bias propagated downstream.
  • Pretext task mismatched to the downstream task.
  • Underestimating the compute and data needed.

Troubleshooting

  • Poor transfer to the target domain — Continue pre-training on domain data or fine-tune with more in-domain examples.
  • Representations ignore needed features — Try a different objective or augmentations.

Contrasted with

Questions this page answers

  • What is self-supervised learning?
  • How are LLMs trained without labels?
  • What is masked language modelling?
  • What is the difference between self-supervised and unsupervised learning?

Typical goals: understand how foundation models are pre-trained; compare next-token and masked objectives.

Sources and verification

Status: not independently verified. This entry was authored on 2026-10-06 from general technical knowledge, without live checking of the sources below. Treat every source as a lead to confirm (identifier, title, authorship, current version) before publication or citation.

Freshness class: stable — Core ideas that change slowly (maths, classic algorithms). Re-review yearly.

Canonical sources (unverified leads):

  • Devlin et al., BERT (2018) — arXiv:1810.04805
  • He et al., Masked Autoencoders Are Scalable Vision Learners (2021) — arXiv:2111.06377
  • Radford et al., Language Models are Unsupervised Multitask Learners (GPT-2, 2019) — locator to be identified