Stanford CS25: Transformers United V6 I From Next-Token Prediction to Next-Generation Intelligence

Two-phase pre-training maximizes data potential

A curriculum approach where phase one emphasizes diverse, lower-quality web data and phase two focuses on high-quality sources (math, code, Wikipedia) with repeated epochs yields 17% better performance than naive training and 3.4% better than optimal blending without ordering.

Quality and epoch estimation prevent diminishing returns

Building quality classifiers and estimating optimal repetition rates for each data source before creating blends ensures high-value tokens are fully utilized without overfitting, addressing projections that LLMs will exhaust 95% of human-generated data by 2030.

Reasoning should be a foundation, not a post-hoc addition

Injecting reasoning data during pre-training creates a 'reason base' model, unlike the standard pipeline that treats reasoning as a post-training fix; this approach yields 16% better performance immediately after pre-training and 9.3% better after SFT.

High-quality pre-training data unlocks hidden post-training gains

While mixing high-quality (SHQ) and low-quality (LDQ) reasoning data showed no benefit immediately after pre-training, models with both achieved a 4.25% boost after SFT, indicating early exposure to quality reasoning primes the model for better refinement later.

Early reasoning advantages are durable and compute-efficient

Models trained with reasoning data during pre-training maintained a 19% average advantage (39% on complex math benchmarks like AIME) after full post-training, and could not be matched by doubling SFT compute or reallocating all reasoning data to post-training.