Stanford CS336 Language Modeling from Scratch | Spring 2026 | Lecture 10: Inference

Inference costs dwarf training rapidly

OpenAI generates 8.6 trillion tokens daily, processing more compute in under four days than DeepSeek v4's entire 32-trillion-token training run.

Agentic workloads remove throughput ceilings

Unlike chatbots constrained by human reading speeds, AI agents generate tokens for internal reasoning and tool use, creating unlimited demand for inference compute.

Three distinct performance metrics matter

Time to First Token (TTFT) drives user perception in interactive apps, latency measures individual streaming speed, and throughput measures batch processing efficiency.

Autoregressive generation prevents parallelization

Training processes all sequence tokens simultaneously, but inference generates one token at a time, eliminating the sequence dimension as a parallelization opportunity.

Small batches make GPUs memory-bound

Matrix operations with batch size B=1 achieve arithmetic intensity near 1, while H100 GPUs require intensity greater than 295 to be compute-bound.

Hardware intensity mismatch defines bottlenecks

When arithmetic intensity falls below the accelerator's theoretical ratio, the system becomes memory-bandwidth limited, which is the default state for inference.

Notation defines tensor dimensions precisely

B=batch size, T=sequence length, D=model dimension, H=head dimension, F=4D (MLP expansion), with S denoting input tokens and T denoting output tokens.

Group Query Attention reduces KV heads

The architecture splits N query heads into K key-value groups with G heads per group, reducing memory for cached keys and values.

Naive inference scales cubically with length

Without caching, generating T tokens requires O(T²) attention computation per step and O(T³) total time.