The "secret sauce" of recent AI breakthroughs: Post-training with RLVR (and RLHF) | Lex Fridman

Verifiable rewards replace learned preferences

Unlike RLHF which optimizes for aggregate human preferences, RLVR uses automatically verifiable outcomes—such as math accuracy or code correctness—as direct rewards, enabling optimization at much larger scales without human labeling.

The generate-grade loop mechanism

Models generate multiple solution attempts and receive rewards based on objective accuracy, creating a reinforcement learning environment where the model learns from trial-and-error on verifiable tasks without human intervention.

Expanding beyond deterministic domains

While math and coding provide clear verification signals, researchers are extending RLVR to open-ended domains using 'rubrics' or LM-as-a-judge frameworks to evaluate more subjective scientific and reasoning problems.

Spontaneous 'aha moments' and self-correction

RLVR training causes models to naturally develop step-by-step reasoning chains and recognize their own errors mid-generation, spontaneously verbalizing mistakes ('Wait, I need to reconsider') without explicit programming of this behavior.

Inference time scaling emerges naturally

Models trained with RLVR inherently learn to use more tokens and 'think longer' to solve problems, with response lengths growing naturally during training as the model discovers that extended reasoning improves final accuracy.

Unlocking existing knowledge versus learning

Experiments show RLVR can rapidly improve accuracy (e.g., 15% to 50% in 50 training steps) by unlocking capabilities already present from pre-training rather than teaching fundamentally new mathematical knowledge.

Post-training compute rivals pre-training

Modern RLVR runs approach pre-training in total GPU hours—Grok 4 reportedly used similar compute for post-training as pre-training—though RLVR is memory-bound (due to generating long sequences) rather than compute-bound like pre-training.

RLVR follows scaling laws; RLHF plateaus

RLVR demonstrates predictable scaling where logarithmic increases in training compute yield linear performance gains, whereas RLHF quickly reaches diminishing returns since preference tuning merely averages stylistic differences rather than solving harder problems.

The three-stage training recipe

Effective reasoning models require sequential intervention: mid-training with curated reasoning traces to establish skills, RLVR for intensive trial-and-error learning on verifiable problems, and RLHF as a final polish for style and tone.

Data contamination debates cloud benchmarks

Researchers debate whether RLVR gains reflect true reasoning or memorization, citing cases like Quen-3 where models achieve suspiciously high precision on specific benchmarks, suggesting potential exposure to similar problems during pre-training.

The path to RLVR 2.0

Future iterations will incorporate process reward models and value functions to grade intermediate reasoning steps rather than just final answers, moving beyond simple question-answer verification to optimize the quality of the thinking process itself.