What is it?
Transformers are built in three layouts. Encoders read a whole text and produce a rich understanding of it (great for search and classification). Decoders write text one token at a time (the chat models). Encoder-decoders read one text and write another (translation, summarisation).
Also known as: encoder vs decoder, BERT vs GPT, decoder-only, encoder-only, seq2seq transformer, T5, causal language model, bidirectional model.
Technical definition
| Encoder-only | Decoder-only | Encoder-decoder | |
|---|---|---|---|
| Attention | Bidirectional | Causal (left-to-right) | Bidirectional encoder + causal decoder with cross-attention |
| Pre-training | Masked LM | Next-token prediction | Span corruption / denoising |
| Typical use | Embeddings, classification, retrieval | Chat, code, general generation | Translation, summarisation, structured transforms |
| Examples | BERT family | GPT-style LLMs | T5, original Transformer |
Decoder-only models dominate current general-purpose LLMs partly for scaling simplicity and in-context learning; the other layouts remain strong for specific tasks.
Entity type: concept · Freshness class: stable.
When should I use it?
Pick the layout matching the task; default to decoder-only for open-ended generation.
Implementation patterns
- Encoder for classification/embedding; decoder for generation.
- Encoder-decoder for translation/summarisation where input and output are distinct sequences.
Technologies and frameworks
Commonly associated: BERT, T5, GPT-style decoders, Hugging Face Transformers. Names are examples of what exists, not recommendations, and projects change quickly.
Practical guide
- Classify task: understand, generate or transform.
- Pick the layout.
- Choose a pre-trained model of that type.
- Compare cost and quality.
Common failure modes
- Using a generative LLM where a small encoder would be cheaper and sufficient.
- Expecting BERT-style encoders to generate fluent text.
Troubleshooting
- Classification is slow/expensive — Try a fine-tuned small encoder.
Contrasted with
- What are Embeddings in AI? — Embedding models are usually encoder-style.
Questions this page answers
- What is the difference between BERT and GPT?
- What is a decoder-only model?
- Which transformer type is best for classification?
- What is an encoder-decoder model?
Typical goals: pick a transformer type for a task; understand why embedding models differ from chat models.
Sources and verification
Status: not independently verified. This entry was authored on 2026-10-06 from general technical knowledge, without live checking of the sources below. Treat every source as a lead to confirm (identifier, title, authorship, current version) before publication or citation.
Freshness class: stable — Core ideas that change slowly (maths, classic algorithms). Re-review yearly.
Canonical sources (unverified leads):
- Vaswani et al., Attention Is All You Need (2017) — arXiv:1706.03762
- Devlin et al., BERT (2018) — arXiv:1810.04805
- Raffel et al., Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer (T5) (2019) — arXiv:1910.10683