EXPLAINER

Distributed Training (Data, Tensor and Pipeline Parallelism)

techniquefreshness: evolvingsources not yet verified

Large models are trained across many accelerators by splitting data, model layers or tensors, and sharding optimiser state. Each strategy trades memory, communication and complexity.

Short answer: Data parallel replicates the model and splits batches; tensor/pipeline parallel split the model; sharding (ZeRO/FSDP) splits parameters, gradients and optimiser state. Large runs combine several, and communication bandwidth often limits speed.

What is it?

A big model can't be trained on one chip, so the work is divided: either each chip gets different data, or each chip holds different parts of the model — often both. The trick is keeping them in sync without wasting time on communication.

Also known as: DDP, FSDP, ZeRO, DeepSpeed, Megatron-LM, tensor parallelism, pipeline parallelism, data parallelism.

Technical definition

Strategies: distributed data parallel (DDP) with gradient all-reduce; ZeRO (Rajbhandari et al., 2019) and PyTorch FSDP shard optimiser state/gradients/parameters; tensor parallelism splits matrix operations (Megatron-LM, Shoeybi et al., 2019); pipeline parallelism splits layers into stages with micro-batching; expert parallelism for Mixture of Experts (MoE). Practical concerns: interconnect bandwidth, checkpointing, fault tolerance, mixed precision and reproducibility.

Entity type: technique · Freshness class: evolving.

When should I use it?

Relevant for training/fine-tuning beyond single-GPU memory.

Implementation patterns

  • Start with DDP; add FSDP/ZeRO when memory-bound.
  • Gradient accumulation to emulate large batches.
  • Frequent checkpoints with resumability.
  • Profile communication overhead.

Technologies and frameworks

Commonly associated: PyTorch DDP/FSDP, DeepSpeed, Megatron-LM, Ray Train, NCCL. Names are examples of what exists, not recommendations, and projects change quickly.

Practical guide

  1. Fit one GPU first.
  2. Move to DDP.
  3. Add sharding if memory-bound.
  4. Profile and tune.
  5. Checkpoint and test resume.

Common failure modes

  • Communication-bound scaling.
  • Non-deterministic divergence across ranks.
  • Checkpoint incompatibility between parallelism setups.
  • Straggler or failed nodes halting jobs.

Troubleshooting

  • Poor scaling beyond a few nodes — Profile all-reduce time; check network topology and batch size.
  • Job dies mid-run — Use frequent checkpoints and elastic/fault-tolerant launchers.

Contrasted with

Questions this page answers

  • What is data parallelism?
  • What is FSDP?
  • What is ZeRO?
  • How do I fine-tune a model on multiple GPUs?
  • What is tensor parallelism?

Typical goals: train or fine-tune across multiple GPUs; understand how frontier models are trained.

Sources and verification

Status: not independently verified. This entry was authored on 2026-10-06 from general technical knowledge, without live checking of the sources below. Treat every source as a lead to confirm (identifier, title, authorship, current version) before publication or citation.

Freshness class: evolving — Established techniques whose best practice shifts as research and tooling move. Re-review every 6 months.

Canonical sources (unverified leads):

  • Rajbhandari et al., ZeRO: Memory Optimizations Toward Training Trillion Parameter Models (2019) — arXiv:1910.02054
  • Shoeybi et al., Megatron-LM (2019) — arXiv:1909.08053
  • PyTorch distributed documentation — pytorch.org/docs/stable/distributed.html