Stanford CS336 Language Modeling from Scratch | Spring 2026 | Lecture 16: Post-Training - RLVR

RLHF hits an overoptimization wall

Training against learned reward models inevitably overfits to preference data, creating a hard annotation bottleneck that limits how much compute can be applied.

Verifiable rewards unlock unlimited scaling

Hard verification signals like math correctness or code execution provide ground-truth objectives similar to AlphaGo, allowing RL to scale indefinitely without overfitting.

Breakthroughs in thinking models

OpenAI's recent solutions to open math problems demonstrate how RLVR enables extended reasoning chains through long-context training on verifiable objectives.

Complexity and sensitivity

PPO requires navigating dozens of implementation details and hyperparameters, with small engineering choices drastically altering optimization outcomes.

The value network burden

PPO requires training a separate value model as large as the policy itself, consuming significant memory that could otherwise support larger models or inference.

Common degenerate configurations

Many practitioners unknowingly reduce PPO to a bandit algorithm by setting gamma=lambda=1, destroying the temporal structure the algorithm was designed to capture.

Eliminating the value network

GRPO removes PPO's most complex component by estimating advantages as z-scores across groups of outputs sampled from the same prompt.

Group-based relative advantage

Instead of comparing rewards to a learned value function, GRPO computes relative performance by comparing each output against the mean and standard deviation of its peer group.

The open-source standard

Originally introduced by DeepSeek, GRPO has become the dominant algorithm for post-training on verifiable tasks due to its simplicity and reduced memory requirements.