The specific engineering that turns "our 70B-parameter model doesn\'t fit on any single GPU" into "we\'re training it across 512 H100s using tensor parallelism within each 8-GPU node (NVLink-connected), pipeline parallelism across node groups, and data parallelism at the outer level — with FSDP sharding optimizer state, gradients, and parameters." Three primary parallelism strategies: Data parallel (DDP for models that fit; FSDP for larger), Tensor parallel (Megatron-LM sharding matrix multiplies within tightly-coupled nodes), Pipeline parallel (GPipe / 1F1B micro-batched stages across nodes). Plus ZeRO/FSDP for memory efficiency and MoE for expert parallelism. Understanding when each strategy fits, how to compose 3D parallelism at frontier scale, and the specific bandwidth + memory constraints that dictate topology is Expert-tier competence for modern LLM training infrastructure.
Distributed ML training is not "just parallel computing applied to gradient descent" — it\'s a specific engineering problem determined by three tight constraints: memory (GPU HBM is limited to 80GB on H100), compute (matmul FLOPs must be kept busy or the accelerator sits idle), and bandwidth (all-reduce, all-gather, reduce-scatter operations across NVLink/PCIe/InfiniBand determine what\'s achievable). Consider concretely what training a modern foundation model requires. A 70B-parameter model in FP16 (2 bytes per param) requires 140GB just for parameters. Add optimizer state (Adam requires 2× FP32 for momentum + variance = 8 bytes per param = 560GB more) and gradients (2 bytes per param = 140GB) — total ~840GB of GPU memory just to hold the training state. Add activation memory for the forward pass (grows with batch × sequence × hidden dimension: often 100-500GB more). A single H100 has 80GB of HBM. The 70B model doesn\'t fit on any single GPU, doesn\'t fit on any node of 8 H100s (640GB total), and requires distributing across multiple nodes — while keeping communication overhead low enough that GPUs actually stay busy computing rather than waiting for data. Every additional device introduces communication cost that eats into throughput. The specific engineering task: decide how to shard the model across devices such that (a) it fits in memory, (b) communication overhead is minimized, (c) compute stays busy, and (d) statistical efficiency is preserved (larger effective batch sizes complicate convergence). This is not a research question — it\'s specific systems engineering. The primitives are: data parallelism (each device has full model, shards data batches, all-reduce gradients), tensor parallelism (split matrix multiplies across devices, all-gather activations), pipeline parallelism (split model into stages across devices, micro-batch through pipeline). ZeRO/FSDP add memory-efficient overlay. At frontier scale (100B+ MoE), 3D parallelism composes all three plus expert parallelism. Understanding the specific mechanisms — and when each fits — is Expert-tier competence.
The specific engineering task M.64 addresses is understanding how the three parallelism primitives compose at different model scales, why each has specific communication + bandwidth requirements, and how to match parallelism topology to hardware topology. The critical insight: distributed training is not "add more GPUs and hope it scales." At every scale, there\'s a specific composition that fits (a) memory budget, (b) compute utilization target (typically 40-60% MFU — Model FLOPs Utilization), (c) communication overhead ceiling, and (d) statistical efficiency preservation (larger effective batch sizes require careful learning rate + warmup adjustments to preserve convergence). Modern distributed training has three primary primitives + memory-efficient overlays: (a) Data parallelism — the workhorse. Each GPU has full model copy. Data batch sharded across GPUs (each processes different subset). Forward + backward computed independently. Gradients synchronized via all-reduce (typically ring all-reduce, Horovod-style, or NVIDIA NCCL AllReduce). Standard PyTorch DDP. Works when model fits on one GPU. Limitation: model + optimizer state must fit per GPU. FSDP (PyTorch) or DeepSpeed ZeRO Stage 3 shard parameters + gradients + optimizer state across DP ranks, enabling larger models. (b) Tensor parallelism — Megatron-LM contribution (NVIDIA, 2019). Split individual layer computations across devices. Specifically for transformer: attention heads split across TP ranks; MLP hidden dimension split across TP ranks. Communication: all-gather activations before layer output; reduce-scatter gradients on backward. Requires high-bandwidth interconnect: NVLink 900 GB/s within node handles it; PCIe (128 GB/s) or InfiniBand (50 GB/s in bytes) too slow. TP degree typically = number of GPUs within a node (e.g., TP=8 for 8-H100 node). (c) Pipeline parallelism — split model into stages. Layers 0-9 on GPU set A; layers 10-19 on GPU set B; etc. Forward pass: input flows through stages sequentially; backward pass: gradients flow reverse. Naive pipeline: massive idle time (bubble). Micro-batching solution (GPipe): split batch into micro-batches, pipeline them through stages, overlapping forward + backward. PipeDream 1F1B (Microsoft): once pipeline fills, alternate 1-forward-1-backward per stage, minimizing memory + bubble time. Cross-node friendly: only send activations at stage boundaries, low bandwidth needed. (d) Memory-efficient overlays: ZeRO stages 1/2/3 (Microsoft DeepSpeed) and FSDP (PyTorch, ZeRO-3 native). Shard optimizer state (Stage 1) / + gradients (Stage 2) / + parameters (Stage 3) across data-parallel ranks. Trade-off: more communication (all-gather params before forward, reduce-scatter grads after) for less memory. Standard for training models that don\'t fit even with TP+PP. (e) 3D parallelism: compose DP × TP × PP for frontier-scale models. Megatron-DeepSpeed reference architecture: TP=8 within node, PP=varies across node groups, DP=N replicas. Each dimension optimized for its constraint (TP for tight-coupling, PP for cross-node scaling, DP for throughput). Standard modern LLM training pattern. Understanding these primitives + their composition is Expert-tier competence.
Each earlier attempt fails specifically. Single-GPU + accumulation fails past model-fit ceiling. Pure DDP requires model fits per device; fails for 13B+ without FSDP. Pure TP across nodes saturates InfiniBand; MFU crashes to 5-15%. The Expert pattern: compose 3D parallelism (TP × PP × DP) matched to hardware bandwidth topology. TP within node (NVLink-tight). PP across node groups (InfiniBand-tolerable). DP across pipeline replicas (throughput). FSDP overlay for memory efficiency. FP8 on H100 for 2× throughput. MoE + expert parallelism for frontier-scale sparse models. Understanding the specific composition — memory math, bandwidth topology, MFU targets, straggler mitigation — is Expert-tier competence for modern LLM training. §02 covers the three parallelism primitives + ZeRO/FSDP mechanisms. §03 covers 3D composition + framework landscape (PyTorch DDP/FSDP, Megatron-LM, DeepSpeed, JAX pjit).
The historical arc of distributed ML training traces specifically how the primitives evolved as models grew. 2012: Google DistBelief. First serious distributed neural network training. Parameter server architecture: workers compute gradients on data shards; parameter servers hold model state; async updates. Enabled ImageNet-scale training. Papers by Jeff Dean et al. Standard early pattern. 2014: Downpour SGD + async parameter servers. Extended DistBelief to larger scales. Async updates enabled higher throughput but stale gradients hurt convergence. Trade-off well-studied. 2015-2016: Sync SGD wins; Horovod emerges. Uber\'s Horovod (Alexander Sergeev, 2017) implements ring all-reduce for synchronous gradient averaging. Cleaner semantics than async parameter servers; better convergence; became standard. NCCL from NVIDIA provides GPU-optimized all-reduce. Ring topology minimizes bandwidth per node (each node sends/receives 2×(N-1)/N × gradient once). 2017: Mixed-precision training on V100. NVIDIA V100 introduces Tensor Cores for FP16 matmul. Micikevicius et al. paper "Mixed Precision Training" (Baidu + NVIDIA): FP16 forward + backward, FP32 master weights + optimizer state, loss scaling. 2-3× speedup with minimal accuracy loss. Standard modern practice. 2018: GPipe. Google\'s pipeline parallelism paper (Huang et al.). Split model across devices as pipeline stages; micro-batching hides bubble time. Applied to AmoebaNet + Transformers. Foundation of modern pipeline parallelism. 2019: Megatron-LM. NVIDIA\'s tensor parallelism paper (Shoeybi et al.). Split matrix multiplies across devices for transformer training. GPT-2 scale (1.5B → 8.3B) trained across 512 V100s. Column-parallel + row-parallel linear layers; all-reduce activations. Foundation of modern tensor parallelism. 2020: ZeRO stages 1/2/3 (DeepSpeed). Microsoft\'s memory-optimization paper (Rajbhandari et al.). Sharding optimizer state (Stage 1) / gradients (Stage 2) / parameters (Stage 3) across data-parallel ranks. Enables training models that don\'t fit even with TP+PP. DeepSpeed framework packages this. Foundation of modern memory-efficient training. 2021: PipeDream 1F1B + interleaved schedules. Microsoft\'s pipeline schedule refinements. 1-forward-1-backward reduces pipeline bubble; interleaved schedules reduce activation memory. Standard modern pipeline schedule. 2022: 3D parallelism (Megatron-DeepSpeed) at GPT-3/PaLM scale. NVIDIA + Microsoft collaboration composes TP + PP + DP for 175B-540B parameter training. Standard modern LLM training pattern. Bloom (Hugging Face + BigScience, 176B) trained with this stack. 2023: FSDP (PyTorch native) + JAX pjit. PyTorch FSDP (Fully Sharded Data Parallel) provides ZeRO-3 equivalent as native PyTorch feature. Meta\'s LLaMA-1/2 training details published. JAX pjit + shard_map enables SPMD-style sharding on TPUs and GPUs. Ecosystem matures. 2024: MoE at scale + FP8 on H100. Mixture-of-Experts models emerge as frontier standard: Mixtral 8x7B (Mistral), DeepSeek-V2 (238B total, 21B active), likely GPT-4 architecture. Expert parallelism (EP) added as 4th dimension. H100 Transformer Engine enables FP8 training (2× throughput vs BF16). Standard 2024 discipline. 2025: MoE + expert parallelism + FP8 standard for frontier. Frontier training runs (Anthropic, OpenAI, Google, Meta) use TP+PP+DP+EP composition with FSDP overlay, FP8 forward, larger sparse models. Understanding this arc — how each primitive addressed a specific scaling wall — is Expert-tier competence.
The three primary parallelism primitives — data, tensor, pipeline — are the specific engineering building blocks for distributed training, and each has a specific communication pattern + bandwidth requirement that dictates where it fits in a physical cluster. (a) Data parallelism (DP): each device holds a full copy of the model. Data batch sharded across devices — device 0 processes samples 0-63, device 1 processes 64-127, etc. Forward + backward computed independently per device. Gradients synchronized via all-reduce collective (typically ring all-reduce for bandwidth efficiency; NCCL implementation on GPUs). PyTorch DDP is the canonical implementation. Standard workhorse. Communication cost per step: 2×(N-1)/N × parameter_size for ring all-reduce. Limitation: model + gradients + optimizer state must fit per device. (b) Tensor parallelism (TP): split individual layer computations across devices. For transformer, Megatron-LM shards: (i) attention heads across TP ranks (each device holds subset of heads); (ii) MLP hidden dimension across TP ranks (column-parallel first linear, row-parallel second linear). Communication: all-gather activations before layer output; all-reduce or reduce-scatter on backward. Requires high-bandwidth interconnect: NVLink 900 GB/s within node handles typical TP=8; PCIe (128 GB/s) or InfiniBand (50 GB/s bulk) too slow for per-layer collectives. Foundational for models too large to fit on one device. (c) Pipeline parallelism (PP): split model into sequential stages, each stage on different device set. Forward pass: input flows through stages 0→1→2→...→N; backward: gradients flow N→...→2→1→0. Naive pipeline: massive idle time (bubble). Solution: micro-batching — split batch into micro-batches, pipeline them through stages. GPipe (Google, 2018) uses "all-forward-then-all-backward" schedule. PipeDream 1F1B (Microsoft) alternates 1-forward-1-backward per stage once pipeline fills, minimizing memory + bubble. Interleaved 1F1B (Megatron-LM) further reduces bubble via virtual pipeline stages. Cross-node friendly: only activations at stage boundaries traverse InfiniBand. (d) ZeRO/FSDP overlay: memory-efficient extension of data parallelism. ZeRO-1 shards optimizer state across DP ranks; ZeRO-2 additionally shards gradients; ZeRO-3 additionally shards parameters (all-gather before forward, discard after). FSDP is PyTorch\'s native ZeRO-3 equivalent. Trade-off: more communication (all-gather params before forward, reduce-scatter grads after backward) for less memory per device. Enables training models 4-8× larger than pure DDP allows. Standard modern discipline; each primitive has its specific fit.
Each GPU holds full model. Shard batch. Ring all-reduce gradients (NCCL). Scales up to ~1024 GPUs when model fits. torch.nn.parallel.DistributedDataParallel is canonical. Workhorse.
Split matmuls across devices. Column-parallel + row-parallel linear layers. All-gather activations per layer. Requires NVLink 900 GB/s. TP=8 fills one H100 node. Cross-node = MFU crash.
Split model into stages. Micro-batch through pipeline to overlap. 1F1B minimizes bubble + activation memory. Interleaved 1F1B (Megatron) reduces bubble further. Cross-node friendly.
Shard optimizer state (S1) / gradients (S2) / parameters (S3) across DP ranks. Trade comm for memory. All-gather params before forward. Reduce-scatter grads after backward. 4-8× memory savings.
PyTorch\'s ZeRO-3 equivalent. Fully Sharded Data Parallel. Modern default for large-model training in PyTorch. torch.distributed.fsdp.FullyShardedDataParallel. Standard 2023+ discipline.
Mixture-of-Experts: shard experts across devices. All-to-all routing sends tokens to their assigned experts. Mixtral, DeepSeek pattern. 4th parallelism dimension for sparse frontier models.
The ZeRO/FSDP memory-sharding mechanism (mech items iv/v) is the specific engineering that made training models like LLaMA-70B feasible on modest cluster sizes. Consider the specific mechanics: (a) The problem being solved: in pure DDP, every GPU holds a full copy of model parameters + gradients + optimizer state. For 70B model with Adam: 140GB params + 140GB grads + 560GB optimizer = 840GB per GPU. Impossible. Even 13B in FP16 with Adam: 26 + 26 + 104 = 156GB per GPU, exceeds H100 80GB. Standard failure. (b) ZeRO-1 (optimizer state sharding): partition optimizer state across N DP ranks. Each rank holds 1/N of Adam momentum + variance (the 560GB → 560/N GB per rank). Parameters + gradients still replicated per GPU. During optimizer step: each rank updates its 1/N shard of params + syncs across ranks. Memory saving: massive (optimizer is 4× larger than params for Adam FP32); communication: modest additional all-gather. Standard first-step ZeRO enablement. (c) ZeRO-2 (+ gradient sharding): additionally partition gradients across DP ranks. During backward: each rank produces gradients for its shard only (reduce-scatter instead of all-reduce). Memory saving: additional 140GB → 140/N GB. Communication: no additional overhead vs ZeRO-1 (reduce-scatter is same volume as all-reduce). Standard. (d) ZeRO-3 (+ parameter sharding): additionally partition parameters across DP ranks. Each rank holds 1/N of params. Before forward pass: all-gather full params for current layer (temporary), compute forward, discard. Before backward: all-gather again. Memory saving: additional 140GB → 140/N GB per GPU. Communication: significant additional all-gather traffic — for 70B model, 140GB all-gathered per forward pass + 140GB per backward pass. Trade-off: 4-8× memory savings for 30-50% additional communication time. Standard for models that don\'t fit even with TP+PP. (e) FSDP (PyTorch): native ZeRO-3 as PyTorch API. torch.distributed.fsdp.FullyShardedDataParallel. Wraps model; sharding automatic; overlaps communication with computation where possible. Standard 2023+ PyTorch large-model training pattern. (f) The overlap optimization: modern FSDP/ZeRO implementations aggressively overlap the all-gather of layer N+1\'s params with layer N\'s compute. Effectively hides communication behind compute if bandwidth is sufficient. Standard modern discipline. (g) The composition with TP+PP: FSDP typically wraps the DP dimension only. Model already sharded by TP within node + PP across nodes; FSDP additionally shards optimizer + gradients + parameters across DP dimension. For 70B on 64 H100s: TP=8 × PP=4 × DP=2, FSDP within DP dimension shards remaining state. Composed 3D+ parallelism with memory efficiency. Standard modern LLM training pattern. Understanding this mechanism — and specifically the trade-off between memory savings and communication overhead — is Expert-tier competence.
The ring all-reduce mechanism (foundational to data parallelism) is worth specific attention because it\'s the primitive underlying every distributed training run\'s gradient synchronization. Consider the specific mechanics: (a) The naive approach: parameter server pattern — all workers send gradients to central server; server averages; broadcasts back. Problems: (i) parameter server is bottleneck (all traffic converges there); (ii) requires 2× N × gradient_size bandwidth at server (N sends + N receives); (iii) doesn\'t scale to hundreds of GPUs. Standard pre-2016 failure. (b) Ring all-reduce (Baidu / Uber Horovod): arrange N GPUs in logical ring. Divide gradient tensor into N chunks. In N-1 reduce-scatter rounds: each GPU sends one chunk to next GPU while receiving one chunk from previous, computing partial sums. After reduce-scatter: each GPU has 1/N of final summed gradient. In N-1 all-gather rounds: each GPU sends its shard around ring. After all-gather: every GPU has full summed gradient. Total bandwidth per GPU: 2×(N-1)/N × gradient_size (approximately 2× regardless of N — the efficiency win). (c) NCCL implementation: NVIDIA\'s NCCL library provides GPU-optimized ring all-reduce (and tree all-reduce for smaller messages). Uses NVLink within node, InfiniBand across nodes. Auto-selects topology-aware algorithm (ring for large messages, tree for small). Standard modern implementation. (d) Hierarchical rings: modern implementations use hierarchical all-reduce — ring within node via NVLink, then ring across nodes via InfiniBand. Amortizes cross-node bandwidth (dominant cost) across only node-level participants. Standard optimization. (e) Communication-compute overlap: modern DDP overlaps gradient all-reduce for layer N with backward compute for layer N-1. Hides all-reduce latency behind compute. Standard PyTorch DDP behavior via gradient_as_bucket_view=True and reducer hooks. Effectively hides all-reduce overhead if compute > comm. (f) Gradient accumulation interaction: when accumulating gradients across K micro-batches per optimizer step, all-reduce happens only after K accumulations. Reduces relative all-reduce overhead. Standard modern pattern. (g) Half-precision considerations: FP16/BF16 all-reduce is 2× faster than FP32. Standard in modern mixed-precision training. FP8 all-reduce (H100) is 4× faster. (h) Scaling limits: ring all-reduce scales O(gradient_size) with node count in bandwidth per node (independent of N to first order), but latency scales O(N). At very large scale (10,000+ GPUs), tree-based or double-binary tree algorithms may outperform. Standard modern discipline. Understanding ring all-reduce is Expert-tier competence — it\'s the specific primitive that made scaling training past hundreds of GPUs feasible.
The primary parallelism strategies are not mutually exclusive — at frontier scale, they compose. 3D parallelism (data × tensor × pipeline) is the standard modern recipe for training models past 13B parameters; expert parallelism (MoE) adds a 4th dimension for sparse frontier models. The specific engineering discipline: match parallelism dimensions to hardware bandwidth topology. Modern GPU clusters have three-tier bandwidth: (i) HBM within GPU (3.35 TB/s on H100 — highest), (ii) NVLink within node (900 GB/s on H100 SXM — high, tight coupling), (iii) InfiniBand across nodes (400 Gb/s = 50 GB/s bulk — modest). Each parallelism dimension has specific bandwidth requirements: tensor parallelism (per-layer collectives, high per-op frequency) requires NVLink; pipeline parallelism (per-stage activation passing, low volume) tolerates InfiniBand; data parallelism (per-step gradient all-reduce) tolerates InfiniBand. The composition rule: TP within node, PP across node groups, DP across pipeline replicas. Standard Megatron-DeepSpeed recipe. Framework landscape: PyTorch DDP (data parallel), PyTorch FSDP (native ZeRO-3), Megatron-LM (NVIDIA, tensor + pipeline + MoE), DeepSpeed (Microsoft, ZeRO + pipeline + mixture), JAX pjit + shard_map (Google, SPMD-style sharding for TPU + GPU). Modern training runs typically compose Megatron-LM for TP+PP with DeepSpeed for ZeRO overlay + FSDP for hybrid PyTorch runs. Understanding this composition — dimension by dimension, matched to bandwidth topology — is Expert-tier competence.
Compose TP × PP × DP matched to hardware topology. TP within NVLink node (in-node). PP across node groups (InfiniBand-tolerable). DP for throughput (all-reduce). Standard modern recipe.
DDP for models that fit per device. FSDP (Fully Sharded Data Parallel) for larger models — native ZeRO-3. torch.distributed.fsdp. Standard 2023+ PyTorch training pattern.
Tensor + pipeline parallelism framework. Attention/MLP TP sharding, PipeDream 1F1B interleaved schedule, activation checkpointing, mixed precision. Foundational for LLM training. Actively maintained.
ZeRO stages 1/2/3, pipeline parallelism, mixture-of-experts, ZeRO-Infinity (CPU + NVMe offload). Composes with Megatron for 3D parallelism ("Megatron-DeepSpeed"). Standard modern stack.
SPMD-style sharding on TPU + GPU. GSPMD compiler auto-parallelizes. Google\'s stack for PaLM, Gemini training. Alternative to PyTorch ecosystem; foundational for TPU-native training.
H100 Transformer Engine for FP8 matmul. 2× throughput vs BF16. FP16 master weights + FP32 accumulators for stability. Standard 2024-2025 discipline. NVIDIA TE library integrates.
The frontier-scale MoE composition (mech item iii/vi extension) is where modern LLM training has moved in 2024-2025 — sparse mixture-of-experts models like Mixtral 8x7B, DeepSeek-V2 (238B total / 21B active), likely GPT-4 architecture, and successor Anthropic/Google frontier models. Consider the specific mechanics: (a) The MoE architecture: instead of dense FFN in each transformer block, replace with N experts (typically 8-64). Router network selects top-K experts per token (typically K=2). Each token routed to K experts; experts compute; outputs weighted-summed. Sparsely activated: only K of N experts run per token, so compute scales with active params (21B for DeepSeek-V2) not total params (238B). Enables larger models with similar training compute. (b) Expert parallelism (EP): shard experts across devices. Each device hosts N/EP experts. Token → router → identify assigned experts → route tokens to their expert devices via all-to-all collective → experts compute → all-to-all back. Communication pattern: all-to-all (each device sends to every other, receives from every other). Requires high bandwidth; can span nodes with careful all-to-all optimization. (c) 4D parallelism: EP composes with TP + PP + DP. Standard frontier recipe: DP × TP × PP × EP. Each dimension matched to constraint. Example DeepSeek-V2: TP=8 within node for dense components, EP for experts, PP across nodes, DP for throughput. (d) The all-to-all communication: the specific bottleneck. For each token routed to K experts, each device sends token embedding to K expert devices (roughly N/EP fraction of tokens leaves this device). Cross-node all-to-all is bandwidth-intensive. Modern implementations use hierarchical all-to-all + expert placement heuristics to minimize cross-node traffic. (e) Expert load imbalance: hot experts (popular tokens routed to them) become bottleneck; cold experts idle. Solutions: auxiliary load-balancing loss (during training) + expert capacity + token dropping. Standard modern discipline. (f) FP8 for MoE: even more valuable for MoE since expert matmuls dominate compute; FP8 gives 2× throughput; H100 TE handles precision management. (g) Framework support: Megatron-LM has MoE support; DeepSpeed-MoE; Tutel (Microsoft) for expert routing; ScaleFold. Modern MoE training combines multiple libraries. Understanding this composition — 4D parallelism with careful all-to-all optimization for MoE — is Expert-tier competence for 2024-2025 frontier training. Standard modern discipline for sparse models.
The practical framework choice + composition (mech items ii-v) deserves specific attention because it\'s where engineering decisions actually get made. Consider the specific stack: (a) PyTorch DDP: canonical data parallelism. torch.nn.parallel.DistributedDataParallel. Standard for models that fit per GPU (up to ~7B on H100). Excellent ergonomics; auto-handles all-reduce; overlaps with backward compute. Foundation of every PyTorch distributed training run. (b) PyTorch FSDP: native ZeRO-3 in PyTorch. torch.distributed.fsdp.FullyShardedDataParallel. Modern default for large-model PyTorch training (2023+). Meta uses this for LLaMA. Excellent when TP+PP+DP+FSDP composition works without needing Megatron. (c) Megatron-LM (NVIDIA): original TP + PP framework. GitHub: NVIDIA/Megatron-LM. Actively maintained. Attention/MLP TP sharding, PipeDream 1F1B interleaved pipeline, activation checkpointing, mixed precision, MoE support. Standard for tensor + pipeline parallelism. Used by every major LLM training run. (d) DeepSpeed (Microsoft): ZeRO/FSDP + pipeline + MoE + ZeRO-Infinity (offload optimizer to CPU/NVMe for extreme memory efficiency). GitHub: microsoft/DeepSpeed. Composes with Megatron ("Megatron-DeepSpeed" reference implementation for GPT-3 scale). Standard for memory-efficient training. (e) Megatron-DeepSpeed: reference composition of both — Megatron for TP+PP, DeepSpeed for ZeRO overlay. Used for BLOOM (176B), GPT-NeoX, many open frontier training runs. Standard modern stack for large-scale PyTorch training. (f) JAX pjit + shard_map: Google\'s SPMD approach. Program written as single-device code; sharding annotations via pjit; XLA compiler auto-parallelizes across devices. Foundational for TPU training (PaLM, Gemini). Also runs on GPU. Alternative ecosystem to PyTorch. (g) Framework choice heuristic: PyTorch for most workloads (broader ecosystem, larger community); FSDP if it works alone (simpler); Megatron-DeepSpeed for frontier-scale (composed TP+PP+ZeRO); JAX for Google-scale TPU workloads. Standard modern discipline. (h) Interoperability: modern frameworks support checkpoint interchange (safetensors), inference-time model loading (HF Transformers), evaluation harnesses (lm-eval-harness). Standard modern engineering. Understanding this framework landscape — and specifically choosing the right composition for your workload + hardware + team — is Expert-tier competence for modern LLM training infrastructure.
Below: each of three parallelism strategies (Data Parallel (DDP/FSDP) · Tensor Parallel (Megatron) · Pipeline Parallel (GPipe/1F1B)) evaluated against three model scales (7B params (single-node) · 70B params (multi-node required) · 1T+ params MoE (frontier scale)). Watch how each strategy fits or fails each scale — Data parallelism (specifically DDP → FSDP) dominates single-node, Tensor parallelism becomes essential within nodes past 13B, Pipeline parallelism becomes essential across nodes past ~70B, and 3D composition (all three) is required at frontier scale. The off-diagonals show where strategy-scale mismatch produces measurably worse MFU or OOM. The takeaway: match parallelism topology to hardware bandwidth topology; compose at frontier scale.
The failure modes of distributed training are specific mechanisms by which "we\'re training on 512 H100s" turns into "MFU is 8% and we\'re paying $500K/day for idle GPUs." Each anti-pattern is a real production pattern; Expert engineers avoid them by matching parallelism topology to hardware bandwidth topology, tracking MFU aggressively, understanding memory math, and choosing FSDP/ZeRO stages appropriately. Recognizing these saves months of "why is our training so slow" debugging.
Pure DDP requires the full model + gradients + optimizer state to fit on every GPU. For 30B with Adam: 60GB params + 60GB grads + 240GB optimizer = 360GB per GPU. H100 has 80GB. Impossible with DDP. The specific fix is FSDP (PyTorch native ZeRO-3) or DeepSpeed ZeRO Stage 3, which shard parameters + gradients + optimizer state across data-parallel ranks. Specifically: (a) THE MEMORY MATH. 30B model in FP16 with Adam FP32 optimizer: (i) parameters: 30B × 2 bytes = 60GB; (ii) gradients: 30B × 2 bytes = 60GB; (iii) Adam optimizer state (m + v in FP32): 30B × 8 bytes = 240GB; (iv) activations: batch × sequence × hidden × layers, typically 50-200GB depending on batch size. Total: 400-600GB per GPU. H100 has 80GB. 5-8× shortfall. DDP cannot fit this. (b) THE FSDP/ZERO-3 FIX. Shard parameters + gradients + optimizer state across the data-parallel dimension: (i) each of N DP ranks holds 1/N of parameters (30GB/N), gradients (30GB/N), optimizer state (240GB/N); (ii) before forward pass: all-gather full parameters for the layer being computed, do forward, discard; (iii) before backward: all-gather again; (iv) after backward: reduce-scatter gradients so each rank has 1/N of gradient; (v) optimizer step: each rank updates its 1/N shard. Memory saving: 8× if N=8. For 30B model with N=64 DP: (60+60+240)/64 + activations ≈ 5-15GB training state per GPU. Fits easily. (c) THE COMMUNICATION COST. FSDP adds all-gather traffic (roughly parameter_size per forward + parameter_size per backward). Trade: 30-50% additional communication for 4-8× memory savings. Standard tradeoff. Modern FSDP overlaps all-gather of layer N+1 with compute of layer N — hides most of the comm cost when compute > comm bandwidth. Standard modern implementation. (d) THE PYTORCH FSDP API. from torch.distributed.fsdp import FullyShardedDataParallel as FSDP; model = FSDP(model, sharding_strategy=ShardingStrategy.FULL_SHARD). Config: (i) sharding_strategy (FULL_SHARD = ZeRO-3, SHARD_GRAD_OP = ZeRO-2, NO_SHARD = ZeRO-1); (ii) auto_wrap_policy for granular sharding; (iii) mixed_precision for BF16/FP16 params + FP32 optimizer; (iv) activation_checkpointing overlay. Standard 2023+ PyTorch large-model training. (e) THE DEEPSPEED ALTERNATIVE. deepspeed --deepspeed_config ds_config.json with ZeRO stage 3 config. Similar semantics; more configurable; integrates with Megatron for TP+PP composition. (f) THE 3D COMPOSITION. For 30B model, likely composition: TP=8 within node (Megatron sharding), FSDP within DP dimension for additional memory savings, no PP needed (30B fits in one pipeline stage after TP sharding). For 70B+: add PP. Standard modern pattern. Understanding this fix — that FSDP/ZeRO-3 is the specific answer to "model doesn\'t fit per GPU with DDP" — is Expert-tier competence. Anti-pattern §05.i captures the failure to shift to FSDP for large models.
Tensor parallelism requires per-layer collectives (all-gather activations, reduce-scatter gradients) that must fit within the interconnect bandwidth budget. NVLink 900 GB/s in-node handles typical TP=8. InfiniBand 50 GB/s bulk cannot — TP collectives across InfiniBand kill MFU. The specific fix is keeping TP within NVLink domain (TP ≤ 8 for H100 nodes) and using PP + DP for cross-node scaling. Specifically: (a) THE ROOT CAUSE. TP is bandwidth-intensive because it requires collective operations PER LAYER, not per step. For 80-layer model: 160 all-gather + reduce-scatter operations per forward+backward pass. Volume per collective: batch × sequence × hidden_dim × 2 bytes (FP16 activations). For batch=8, seq=4096, hidden=8192: 512MB per collective. Standard sizes. (b) THE BANDWIDTH MATH. NVLink 900 GB/s: 512MB collective takes ~0.5ms. Total per-layer collective overhead: 160 × 0.5ms = 80ms per step. Compute per step: ~200-500ms typical. Comm/compute ratio: 20-40%. MFU: 40-60% (respectable). InfiniBand 50 GB/s: 512MB collective takes 10ms. Total: 160 × 10ms = 1600ms per step. Compute: same 200-500ms. Comm/compute ratio: 300-800%. GPUs sit idle 80% of the time. MFU: 5-15% (catastrophic). (c) THE FIX: KEEP TP WITHIN NVLINK DOMAIN. For H100 SXM nodes: NVLink connects all 8 GPUs within node. TP degree = 8 fills exactly one NVLink domain. Never span TP across nodes (across InfiniBand). Rule: TP ≤ number of GPUs per NVLink domain. Standard modern discipline. (d) THE CROSS-NODE SCALING. For 70B model on 8-node cluster (64 GPUs): TP=8 within each node (uses NVLink), PP across node groups (e.g., PP=4 across 4 node groups, DP=2 replicas). Standard 3D composition. Cross-node communication only for PP activation passing (small volume) and DP gradient all-reduce (per-step, tolerant of InfiniBand). Standard modern pattern; MFU 40-55% achievable. (e) THE EXCEPTION: NVLink Switch systems. NVIDIA NVL72 (H100 with NVLink Switch) provides all-to-all NVLink across 72 GPUs (18 nodes). In this rare topology, TP can span beyond 8. But standard H100 SXM nodes cap TP at 8. Standard modern discipline. (f) THE MEASUREMENT. Track MFU (Model FLOPs Utilization) as primary metric: FLOPs computed / (theoretical peak × devices × time). H100 peak: 989 TFLOPS BF16. If observing MFU < 30% at large scale, comm bottleneck likely. Diagnose with profiling (nsys, PyTorch profiler); typical culprit is cross-node TP or unbalanced PP schedule. Standard modern discipline. Understanding TP\'s bandwidth requirement — and matching it to NVLink topology — is Expert-tier competence. Anti-pattern §05.ii captures the failure to respect topology.
Naive pipeline parallelism (all-forward-then-all-backward) has massive bubble time — only one stage is active at any moment. The specific fix is micro-batching + 1F1B interleaved schedule, which keeps most stages busy most of the time by overlapping forward/backward across micro-batches. Specifically: (a) THE NAIVE PATTERN. Batch enters stage 0. Stage 0 computes forward, sends activations to stage 1, sits idle. Stage 1 computes forward, sends to stage 2, sits idle. ... Stage N-1 computes forward, computes loss, computes backward for its layers, sends gradients back. ... Stage 0 finishes backward. Only one stage active at any moment. For PP=8: 7/8 of GPUs idle 7/8 of the time. Catastrophic. Standard failure of naive pipeline. (b) THE MICRO-BATCHING FIX (GPipe). Split batch into M micro-batches (typical M=16-64). Pipeline them: micro-batch 0 enters stage 0, then stage 1 while micro-batch 1 enters stage 0, etc. After M steps: pipeline full, all stages active. Bubble fraction: (P-1)/(P+M-1) where P=pipeline depth, M=micro-batches. For P=8, M=32: bubble = 7/39 = 18%. For M=64: bubble = 10%. Standard modern practice; M chosen based on activation memory budget. (c) THE 1F1B SCHEDULE (PipeDream, Microsoft). Once pipeline fills, alternate 1-forward-1-backward per stage. Reduces activation memory (only 1 micro-batch worth of activations retained per stage — vs GPipe M micro-batches). Same bubble as GPipe. Better memory profile. Standard for memory-constrained training. (d) THE INTERLEAVED 1F1B (Megatron-LM). Virtual pipeline stages — each device holds multiple non-contiguous chunks of layers. Cyclic scheduling further reduces bubble time. For P=8, V=4 (virtual chunks per rank), M=32: bubble = 7/(32 × 4 + 7) = 5%. Standard for large-scale LLM training. (e) THE BUBBLE-BANDWIDTH TRADEOFF. More micro-batches → less bubble but more activation memory + more per-micro-batch overhead. Sweet spot: M = 4-8 × P typical. Standard tuning. (f) THE MEGATRON API. --pipeline-model-parallel-size 8 --num-layers-per-virtual-pipeline-stage 5 configures interleaved 1F1B. Standard modern config. DeepSpeed has similar config. (g) THE MEASUREMENT. Pipeline utilization visualization: track per-GPU active vs idle time. Modern profilers (nsys, PyTorch profiler) show pipeline bubbles clearly. Target: < 10% bubble. Alerts when bubble > 20%. Standard modern discipline. Understanding this fix — that naive pipeline fails and micro-batching + 1F1B interleaved is standard — is Expert-tier competence. Anti-pattern §05.iii captures the failure to use proper pipeline schedule.
Modern distributed training frameworks aggressively overlap communication with computation — starting the all-gather for layer N+1 while computing layer N. Without overlap, GPUs idle during comm. The specific fix is enabling prefetching + async communication + proper bucket sizing in FSDP/DDP configuration. Specifically: (a) THE PATTERN. Naive FSDP forward: (i) gather layer 0 params (idle GPU), (ii) compute layer 0 (idle comm), (iii) gather layer 1 params (idle GPU), (iv) compute layer 1 (idle comm), ... . 50% overhead compared to serial compute. Standard failure without overlap. (b) THE OVERLAP MECHANISM. Aggressive prefetching: (i) gather layer 0 params, (ii) compute layer 0 WHILE gathering layer 1 params in parallel, (iii) compute layer 1 WHILE gathering layer 2 params, ... . Communication hidden behind compute. If compute time per layer > gather time, comm cost approaches zero. Standard modern implementation. (c) THE FSDP CONFIGURATION. FSDP(model, forward_prefetch=True, backward_prefetch=BackwardPrefetch.BACKWARD_PRE, use_orig_params=True). forward_prefetch enables next-layer gather during current-layer compute. Standard for hiding comm. (d) THE DDP CONFIGURATION. DistributedDataParallel(model, gradient_as_bucket_view=True). DDP buckets gradients — reduces number of small all-reduces (bad — high latency overhead) into fewer larger ones (good — bandwidth-limited). Typical bucket: 25MB. Also enables overlap of gradient all-reduce with backward compute of earlier layers. Standard PyTorch DDP behavior. (e) THE ACTIVATION CHECKPOINTING INTERACTION. Activation checkpointing recomputes forward during backward (trades compute for memory). Interacts with FSDP: with activation checkpointing, backward gather is per-recompute-forward pass. Framework handles interaction; verify configuration correct. Standard consideration. (f) THE ASYNC COMMUNICATION PRIMITIVES. NCCL supports async collectives via CUDA streams. Modern PyTorch uses non-default streams for communication, enabling true overlap with compute stream. Standard modern implementation; verify with profiler. (g) THE MEASUREMENT. Profile with nsys or PyTorch profiler. Look for: (i) compute streams busy > 90% of the time (good); (ii) communication streams overlap compute streams (good); (iii) large gaps where GPU idle waiting for collective (bad — indicates missing overlap). Standard modern discipline. (h) THE INVESTIGATION FLOW. Debug slow FSDP: (i) verify forward_prefetch enabled; (ii) verify communication on separate CUDA stream; (iii) verify bucket size sufficient (small buckets have latency overhead); (iv) check activation checkpointing configuration if using; (v) verify NCCL config uses correct hierarchical all-reduce for cross-node. Standard debugging flow. Understanding this — that comm-compute overlap is default modern behavior and must be verified — is Expert-tier competence. Anti-pattern §05.iv captures the failure to configure overlap.
Modern LLM training runs cost hundreds of thousands to millions of dollars per day. Model FLOPs Utilization (MFU) is the specific metric that measures how efficiently GPU compute is being used. Not measuring MFU means burning money on idle GPUs without knowing. Specifically: (a) THE MFU DEFINITION. Model FLOPs Utilization = actual FLOPs computed for model forward+backward / (theoretical peak FLOPs × device count × time). For H100 BF16: 989 TFLOPS peak per GPU. For 1024 H100s at 100% MFU: 1024 × 989 = ~1 EFLOPS. Actual training throughput measured in tokens/second, converted to FLOPs via model architecture (6N per token for dense transformer, N=params; higher for MoE). MFU = actual_flops / theoretical_peak. Standard modern metric. (b) THE TARGET RANGES. Well-tuned modern LLM training: 40-55% MFU on H100. Frontier labs (Anthropic, OpenAI, Google) report 45-55% MFU as production-typical. Some optimized workloads: 55-60%. Very rarely > 60% (physics limits — memory bandwidth, comm overhead, straggler effects). Below 30% typically indicates specific problem. Below 20% is catastrophic. Standard modern reference. (c) THE COST MATH. H100 SXM in cloud: ~$3-5/hour on-demand, ~$2-3/hour reserved. 1024 GPUs × $3/hour × 24 hours = $73,728/day per unit. 3-4 units for medium training run = $200-300K/day. Frontier scale: $500K-$2M/day. At 15% MFU vs 50% MFU: 3× cost overrun. Real dollars. (d) THE DIAGNOSIS PLAYBOOK. MFU too low? (i) Profile with nsys, PyTorch profiler; (ii) check for comm bottleneck (cross-node TP? large all-reduce?); (iii) check for straggler effects (some GPUs slower — hardware issues, temperature, congestion); (iv) check for I/O bottleneck (data loading not keeping up with compute); (v) check pipeline bubble percentage; (vi) check activation memory pressure (recomputation overhead); (vii) check for missed overlaps (comm not hidden). Standard modern debugging flow. (e) THE MONITORING DISCIPLINE. Modern training runs report MFU per step + rolling average to metrics dashboard (Grafana, Weights & Biases, TensorBoard). Alerts when MFU drops below threshold (e.g., 30%). Post-mortem on any drop. Standard modern discipline. Frontier labs treat MFU as first-class performance metric. (f) THE OPTIMIZATION LOOP. (i) Measure baseline MFU; (ii) profile bottleneck; (iii) fix top bottleneck (e.g., increase micro-batches for less bubble, add FSDP overlap, tune all-reduce buckets); (iv) measure again; (v) iterate. Standard performance engineering discipline. (g) THE HISTORICAL WINS. Some published optimization results: PaLM training achieved 46% MFU (Chowdhery et al., 2022); LLaMA achieved similar. Frontier improvements: FP8 on H100 can achieve 55%+ MFU with Transformer Engine. Standard modern targets. (h) THE ORG DISCIPLINE. Training infrastructure teams have MFU as a first-class KPI. Weekly reviews of top runs; MFU regressions treated as incidents. Cost saved at scale is enormous. Standard modern practice. Understanding MFU — how to measure, optimize, monitor — is Expert-tier competence. Anti-pattern §05.v captures the failure to treat MFU as a first-class metric.
The composite pattern across all five is that distributed training failure modes reflect specific engineering gaps in memory management, bandwidth-topology matching, pipeline scheduling, communication-compute overlap, and utilization monitoring. Pure DDP for models that don\'t fit ignores the memory math. Tensor parallelism across nodes ignores bandwidth topology. Naive pipeline ignores bubble time. Not overlapping comm with compute ignores async execution. Ignoring MFU burns money silently. Each has specific fixes: (a) FSDP/ZeRO-3 for large models; (b) TP within NVLink domain only; (c) micro-batching + 1F1B interleaved schedule; (d) forward_prefetch + async streams; (e) MFU as first-class monitored metric. Getting distributed training right is the specific engineering discipline that turns "our 70B training is unstable and slow" into "we\'re training LLaMA-70B at 50% MFU on 512 H100s with proper 3D parallelism composition, FSDP memory sharding, FP8 mixed precision, and full instrumentation."
The terms that show up in every distributed training design review, every Megatron config discussion, every 3D parallelism debate.
torch.distributed.fsdp.FullyShardedDataParallel. Standard 2023+ PyTorch large-model training. Auto-shards params + grads + optimizer; overlaps comm with compute.Test the distributed training understanding. Click an answer; explanation drops in instantly.
Perfect. 3D parallelism composition, FSDP memory sharding, bandwidth-topology matching, pipeline scheduling, MFU discipline — the specific engineering for modern LLM training infrastructure. Next: M.65.
The composite understanding that turns "our 70B model won\'t fit on any single GPU" into "we\'re training it across 512 H100s with TP=8 within NVLink nodes, PP=8 across InfiniBand groups, DP=8 replicas, FSDP optimizer sharding, and FP8 mixed precision — achieving 50% MFU."
NVLink 900 GB/s in-node → TP fits (per-layer collectives). InfiniBand 50 GB/s cross-node → PP fits (per-stage activations) + DP fits (per-step gradient all-reduce). Never span TP across InfiniBand — MFU crashes to 5-15%. Standard composition: TP within node, PP + DP across nodes. 3D parallelism at frontier scale. Standard modern discipline.
Pure DDP requires model + gradients + optimizer state fit per GPU. For 30B+ with Adam: impossible on H100. FSDP (PyTorch native ZeRO-3) shards params + grads + optimizer across DP ranks; 4-8× memory savings for 30-50% additional communication. Standard 2023+ pattern. Compose with TP+PP at frontier scale. Standard modern discipline.
Modern LLM training runs cost $200K-$2M/day. Model FLOPs Utilization (MFU) measures efficiency. Target: 40-55% on H100. Below 30% indicates specific problem; below 20% catastrophic. Measure per step, alert on drops, post-mortem on regressions. Not tracking MFU means burning money silently. Standard modern discipline.