Stanford CME296 Diffusion & Large Vision Models | Spring 2026 | Lecture 2 - Score matching

Score defined as gradient of log probability

The score is defined as ∇ₓ log p(x), representing the direction of steepest ascent toward regions of higher probability density in the data space.

Log probability eliminates normalizing constants

Unlike ∇p(x) which requires knowing the intractable normalizing constant Z, the gradient of log p(x) eliminates Z entirely since ∇ log Z = 0.

Numerical stability in low-density regions

Dividing the gradient by the probability density (as in ∇ log p = ∇p/p) prevents numerical instability when p(x) takes very small values in sparse regions of the data space.

Following scores leads to high-density regions

Starting from random noise, iteratively following the score direction moves samples toward regions of higher probability under the data distribution.

Stochastic sampling ensures diversity

Langevin sampling adds a stochastic noise term to the score-following process, preventing mode collapse and ensuring exploration of the full distribution rather than just its highest-density points.

MCMC method with theoretical guarantees

Langevin dynamics is a Markov Chain Monte Carlo method derived from the Fokker-Planck equation that converges to the true data distribution when the score is known accurately.

Unknown true score requires approximation

Since the true data distribution p_data is unknown, the score cannot be computed directly, necessitating methods to estimate it without explicit knowledge of the density.

Gaussian perturbations provide tractable targets

Adding Gaussian noise to data creates a perturbed distribution q_σ(x̃|x) with an analytically known score equal to -(x̃ - x)/σ², enabling supervised learning.

L2 loss on score predictions

The model s_θ is trained to minimize the squared error between its predicted score and the true score of the noised data, effectively learning a denoising score function.