Stanford CS336 Language Modeling from Scratch | Spring 2026 | Lecture 11: Scaling Laws

Stabilize learning rates across model scales

MUP ensures the optimal learning rate remains constant (approximately 10^-2) regardless of model size by scaling embedding outputs, residual connections by the square root of layer count, and weight initializations by fan-out ratios.

Implement per-parameter learning rate scaling

Unlike standard training, MUP requires setting different learning rates for specific tensor types such as embeddings, LM heads, and matrix weights rather than using a single global rate.

Enable efficient scaling ladders

Stable hyperparameters allow researchers to train small proxy models and extrapolate directly to target sizes with 5x parameter gaps, eliminating the need to brute-force tune large models.

Optimal batch size scales with target loss

Following Kaplan's critical batch size analysis, experiments show optimal batch size follows a power law with respect to the loss target, where achieving lower loss requires proportionally larger batch sizes.

Derive batch size from loss targets

Researchers can precisely set batch sizes using scaling curves that map specific loss targets to their corresponding optimal batch sizes, balancing compute efficiency against sample efficiency.

Enable training extension without restart

The trapezoidal WSD schedule maintains a constant learning rate during a long stable phase comprising 80-90% of training, allowing researchers to rewind to any stable checkpoint and extend training rather than restarting from scratch.

Concentrate gains in rapid decay phase

The final 10-20% of training uses rapid decay to approximately 10% of the maximum learning rate, capturing significant loss improvements that match or exceed cosine schedule performance.

Simplify Chinchilla scaling analyses

WSD eliminates the quadratic cost of traditional Chinchilla isoflops experiments by allowing a single long run to serve multiple data scaling analyses through checkpoint rewinding and re-decay.