Stanford CS336 Language Modeling from Scratch | Spring 2026 | Lecture 9: Scaling Laws

Power law scaling predates modern language models

Research from Bell Labs (1993) and Hestness et al. (2017) established that model error decays polynomially with dataset size across diverse domains including speech recognition and machine translation.

Scaling laws parallel classical generalization theory

These laws function as empirical measurements of sample complexity, directly paralleling classical generalization bounds that theoretically predict performance changes as training data grows.

Data scaling advantages documented in early NLP

Banko and Brill demonstrated that increasing data often outperforms algorithmic improvements, while Kachina et al. (2012) confirmed that power laws fit machine translation blue score improvements.

Optimize hyperparameters on small scale experiments first

Practitioners should tune all architectural choices and hyperparameters on small, cheap experiments rather than risking millions of dollars on direct large-scale training runs.

Extrapolate small model performance to large scale

Power-law relationships appear as linear trends in log-log plots, providing robust mathematical rules to extrapolate small-model behavior to large-scale compute and parameter regimes.

Scaling laws enable confident frontier model planning

This paradigm serves as an engineering "way of life" that allows confident planning of frontier model training through predictable regularities between resources and loss.

Test loss follows power law decay patterns

When plotted against dataset size, compute, or parameters, test loss follows power laws appearing as linear decay in log-log space far from the irreducible error floor.

Different metrics show distinct scaling law forms

While pre-training loss follows power laws, downstream benchmark accuracy typically follows sigmoid curves, and capability improvements show linear trends against deployment dates.

Power laws break down near irreducible error

These polynomial trends hold reliably when models operate far from the asymptotic entropy limit but break down as performance approaches the theoretical noise ceiling.