Stanford CS336 Language Modeling from Scratch | Spring 2026 | Guest Lecture: Dan Fu

Dramatic parameter growth across generations

Language models scaled from 100 million parameters in 2018 to over 1 trillion today, with frontier models reaching 5-10 trillion parameters.

Rapid societal transition to AI adoption

AI adoption mirrors the 1902-1912 shift from horses to cars in Manhattan, with 2024 marking the inflection point where developers began writing majority of code using AI assistants.

Inference as the intelligence engine

While GPUs are the 'new oil,' inference engines and GPU kernels serve as the actual mechanism converting electricity into usable tokens and intelligence.

Complex end-to-end request lifecycle

Each request undergoes scheduling, KV cache lookup for computation reuse, distributed execution across GPUs, and safety checks before token output.

Prefill versus decode phase bottlenecks

Prefill processes input tokens in parallel and is compute-bound, while decode generates outputs sequentially and is memory bandwidth-bound.

Diverse real-world workload characteristics

Coding agents process tens of thousands of input tokens with brief outputs, while chat and summarization tasks exhibit radically different computational patterns.

Continuous batching maximizes GPU utilization

Modern systems employ continuous batching to dynamically mix requests of varying lengths, processing short and long generations simultaneously to maximize throughput.

Navigating complex agentic workflow patterns

Multi-turn conversations with variable gaps between interactions, tool use iterations, and diverse latency requirements create traffic patterns distinct from training workloads.