Stanford CME296 Diffusion & Large Vision Models | Spring 2026 | Lecture 5 - Architectures

Three essential inputs

The generation model processes a noisy latent representation XT, a timestep t indicating noise level, and a condition c (text or image) to guide the generation process toward specific outputs.

Velocity prediction output

The model predicts a velocity vector field (or equivalently noise/score) that must share identical dimensions with the input to satisfy the iterative denoising update equation xt+dt = xt + v*dt.

Dual-scale understanding

Effective architectures must simultaneously capture global image structure for coherence and fine local details for crispness while remaining scalable to high resolutions.

Human-like scanning bias

Convolution operations impose an inductive bias where learnable filters scan across spatial dimensions, extracting local visual features like edges and textures similar to human visual processing.

Receptive field limitations

Standard convolutions have limited receptive fields where early layers see only nearby pixels, preventing global context understanding without prohibitively deep stacking.

Hierarchical downsampling solution

Pooling operations reduce spatial dimensions while exponentially expanding the receptive field, allowing deeper layers to understand global image structure efficiently.

Encoder-decoder structure

The U-Net employs an encoder path (convolutions and pooling) to compress the image into a bottleneck representation with global context, followed by a decoder path using transpose convolutions to restore original dimensions.

Skip connections preserve detail

Direct concatenation of encoder feature maps to corresponding decoder layers transports local details that are lost during downsampling, enabling the generation of crisp, high-fidelity outputs.

Distinction from autoencoders

Unlike VAEs that compress to latent spaces for reconstruction, diffusion U-Nets predict denoising directions (velocity/noise) and must maintain strict input-output dimensional consistency for the iterative sampling process.