Stanford CS336 Language Modeling from Scratch | Spring 2026 | Lecture 7: Parallelism
TL;DR
This lecture introduces distributed training for large language models, explaining how to scale beyond single-GPU memory limits using collective communication primitives (all-reduce, all-gather, reduce-scatter) across hardware topologies ranging from NVLink-connected single nodes to InfiniBand-linked multi-node clusters.
🚀 The Multi-GPU Scaling Imperative 3 insights
Memory constraints drive distribution
Training trillion-parameter models requires multiple GPUs because model weights, activations, and optimizer states exceed single-GPU HBM capacity (e.g., 192GB on NVIDIA B200).
Speed scaling requires trade-offs
Even when models fit on one GPU, distributing computation accelerates training, though practitioners must calculate whether communication bandwidth costs outweigh the benefits of additional compute.
Extended memory hierarchy
The data locality challenge extends from single-GPU registers-to-HBM to a three-tier system: single GPU, NVLink-connected single-node multi-GPU, and InfiniBand/Ethernet-connected multi-node clusters.
🔄 Collective Communication Primitives 4 insights
Abstracting distributed operations
Collective operations provide high-level communication templates across ranks (devices), eliminating the need to manage complex point-to-point GPU transfers manually.
All-reduce for data parallelism
This operation sums gradients across all GPUs and replicates the result to every device, serving as the foundation for basic distributed training implementations.
Decomposed operations for efficiency
Advanced strategies like FSDP break all-reduce into reduce-scatter followed by all-gather, enabling intervention and more efficient management of optimizer states and parameter sharding.
All-to-all for dynamic routing
This general collective enables Mixture-of-Experts training by routing activations between ranks based on data-dependent decisions, functioning as a generalized matrix transpose across unbalanced loads.
🔌 Hardware Topology and Bandwidth 3 insights
Intra-node interconnects
Modern single-node systems utilize NVLink and NVSwitch for high-bandwidth GPU communication, while PCIe represents a legacy bottleneck unsuitable for serious distributed training.
Inter-node networking
Multi-node clusters rely on InfiniBand or standard Ethernet to connect servers, creating significant latency penalties that make minimizing cross-node data transfer critical for performance.
The data movement bottleneck
Effective parallelism requires orchestrating computation to keep data close to compute, treating remote GPU memory as the slowest tier in the hierarchy and avoiding unnecessary shuffles.
Bottom Line
Master collective communication primitives—specifically all-reduce, all-gather, and reduce-scatter—as they form the foundational building blocks for implementing efficient data, tensor, and pipeline parallelism across any GPU cluster.
More from Stanford Online
View all
Stanford Robotics Seminar ENGR319 | Spring 2026 | Towards Trustworthy Autonomy
As learning-based robotics deploy at scale—exemplified by Waymo's 500,000 weekly rides—they face dangerous 'semantic anomalies' where context causes system-level confusion rather than visual novelty. The speaker presents a 'fast and slow' reasoning framework using lightweight embedding models for real-time detection and large language models for safety interventions, enabling trustworthy autonomy without requiring perfect prediction models.
Stanford MS&E435 Economics of the AI Supercycle | Spring 2026 | Applications, Coding AI
Vercel founder Guillermo Rauch explains how AI coding agents have expanded the software development market by 10-100x, driving a fundamental shift from traditional web services to 'agentic infrastructure' where tokens replace pixels as the primary commodity and deployment becomes the critical value creator.
Stanford MS&E435 Economics of the AI Supercycle | Spring 2026 | Building AI Factories
Crusoe Energy CEO Chase Lockmiller explains how AI data centers represent history's second-largest infrastructure investment, driven by the economic potential of scalable 'digital labor.' He reveals Crusoe's strategy of building massive AI factories in stranded-power locations like Abilene, Texas, to overcome the industry's critical bottleneck: energized data center capacity.
AI in Healthcare Series: Inside the Rise of AI in Healthcare, Open Evidence and Cyber Risks
Former U.S. Chief Data Scientist DJ Patil warns that healthcare systems are dangerously unprepared for AI-enabled cyberattacks from nation states, while simultaneously seeing rapid democratization of medical knowledge through tools like Open Evidence that are fundamentally reshaping the doctor-patient relationship.