Stanford CS336 Language Modeling from Scratch | Spring 2026 | Lecture 12: Evaluation

Benchmarks versus cost efficiency

While Artificial Analysis ranks models by intelligence index, cost matters—expensive models don't always justify their price premium when plotted against inference costs.

Human preference rankings

Arena AI (formerly Chatbot Arena) ranks models based on blind user preferences, capturing subjective quality that pure accuracy metrics fail to measure.

Economic usage signals

Open Router statistics reveal which models people actually pay to use, offering a market-driven lens of practical value beyond technical benchmarks.

Zero-shot evaluation paradigm

GPT-2 shifted from in-distribution testing to zero-shot evaluation on standard datasets, establishing the modern practice of measuring generalization rather than memorization.

The scaling hypothesis

The belief that "perplexity is all you need" drives scaling laws research, arguing that minimizing perplexity toward the true distribution's entropy inevitably unlocks AGI capabilities.

Conditional perplexity

Researchers can focus on specific tokens by measuring conditional perplexity given prompts, weighting important predictions like answers over incidental words like articles.

Leaderboard trust issues

Perplexity competitions require trusting submitted probabilities sum to one, unlike downstream tasks where black-box models can be evaluated objectively via outputs.

Controlled academic testing

Exam-style benchmarks provide carefully controlled difficulty and unambiguous correct answers, making them ideal for automated grading of knowledge and reasoning.

MMLU's evolution

The Massive Multitask Language Understanding benchmark originally showed GPT-3 barely above random chance on 57 subjects using few-shot prompting, but has since saturated into the 90s.

Benchmark lifecycles

Successful benchmarks follow a predictable arc from novel challenges demonstrating emergent capabilities to trivial tasks that require creating harder successors.