What is it?
Instead of sliding filters over an image, a Vision Transformer chops it into small squares ("patches"), treats each like a word, and lets attention work out how the patches relate. Given enough training data it performs very well and forms the visual half of many multimodal AI systems.
Also known as: ViT, vision transformer, image transformer, patch embedding, Swin transformer, DeiT, transformers for images.
Technical definition
ViT (Dosovitskiy et al., 2020) linearly embeds fixed-size patches, adds position embeddings and a class token, and applies transformer encoder layers. Lacking CNN inductive biases, it generally needs large datasets or strong augmentation/distillation to match CNNs on smaller data. Variants: Swin (shifted windows), DeiT (data-efficient training), and use as image encoders in Contrastive Learning and CLIP and Vision-Language Models (VLMs). Cost grows quadratically with patch count.
Entity type: model · Freshness class: evolving.
When should I use it?
Use with pre-trained weights for classification, detection and as visual encoders in multimodal systems.
Implementation patterns
- Use a pre-trained ViT backbone with a task head.
- Larger patch size to cut compute.
- Hierarchical/windowed variants for dense tasks.
- ViT as the image encoder in a VLM.
Technologies and frameworks
Commonly associated: Hugging Face Transformers, timm, PyTorch. Names are examples of what exists, not recommendations, and projects change quickly.
Practical guide
- Pick a pre-trained checkpoint.
- Match input resolution/normalisation.
- Fine-tune with small LR and layer-wise decay.
- Compare against a CNN baseline.
Common failure modes
- Overfitting when trained from scratch on small data.
- High memory at high resolution.
- Resolution mismatch with pre-training position embeddings.
- Ignoring licence of pre-trained weights.
Troubleshooting
- Poor accuracy on small dataset — Fine-tune a pre-trained ViT or use a CNN.
- Out of memory — Lower resolution, larger patches, gradient checkpointing, or windowed variants.
Contrasted with
- Convolutional Neural Networks (CNNs) — CNNs build in locality; ViT learns spatial relations from data.
Questions this page answers
- What is a Vision Transformer?
- How does ViT work?
- ViT vs CNN: which is better?
- Why do vision transformers need so much data?
Typical goals: understand how transformers handle images; choose an image backbone.
Sources and verification
Status: not independently verified. This entry was authored on 2026-10-06 from general technical knowledge, without live checking of the sources below. Treat every source as a lead to confirm (identifier, title, authorship, current version) before publication or citation.
Freshness class: evolving — Established techniques whose best practice shifts as research and tooling move. Re-review every 6 months.
Canonical sources (unverified leads):
- Dosovitskiy et al., An Image is Worth 16x16 Words (2020) — arXiv:2010.11929
- Liu et al., Swin Transformer (2021) — arXiv:2103.14030