Stanford CME296 Diffusion & Large Vision Models | Spring 2026 | Lecture 7 - Evaluation

Prompt adherence vs. aesthetic quality

Models must be judged both on following input prompts accurately and on physical plausibility and visual appeal.

Safety, diversity, and bias requirements

Beyond quality and adherence, evaluations must consider safety concerns, output diversity, and bias mitigation to ensure robust deployment.

Absolute rating scales introduce noise

While 1-5 scales offer nuance, they introduce subjective variance as humans struggle to consistently distinguish between adjacent ratings like 4 and 5.

Binary classification simplifies judgment

Reducing judgments to "good" or "bad" simplifies the task but remains challenging on absolute scales without reference points.

Pairwise comparisons minimize subjectivity

Asking humans to choose between two images is cognitively easier and produces more consistent relative judgments than absolute scoring.

Win rates ignore opponent strength

Defeating a weak model should not count equally to defeating a state-of-the-art model, rendering raw win rates misleading.

Dynamic adjustment based on expected outcomes

The system calculates expected win probabilities based on current ratings and updates scores proportionally to the "surprise" factor of actual outcomes.

Efficient leaderboard maintenance

ELO enables dynamic model ranking without requiring exhaustive round-robin comparisons every time a new model enters the evaluation pool.

Human evaluation faces scalability limits

Manual rating is prohibitively expensive, time-consuming, and vulnerable to human inconsistency and fatigue.

Necessity of automated metrics

These limitations create necessity for reference-free automated evaluation methods that can scale with model development cycles.