Lecture 15 - PCA and ICA | Stanford CS229: Machine Learning Andrew Ng - Autumn 2018

Non-probabilistic dimensionality reduction

Unlike Factor Analysis which models P(X), PCA does not build a density model but instead finds the low-dimensional subspace where data varies most significantly.

Principal axis captures true variation

PCA identifies the direction of maximum variance, such as the underlying height factor when measuring children in both inches and centimeters, effectively separating signal from orthogonal noise.

Two equivalent mathematical formulations

The algorithm equivalently minimizes the sum of squared projection distances to the axis or maximizes the variance of the projected data, with both approaches yielding the same solution via the Pythagorean theorem.

Eigenvectors of covariance matrix

The optimal projection direction u is the principal eigenvector of the covariance matrix Σ, where Σ = (1/m) Σ x_i x_i^T for mean-zero data.

Constrained optimization solution

Maximizing u^T Σ u subject to the constraint ||u|| = 1 yields Σu = λu, proving u must be an eigenvector of Σ with the largest eigenvalue indicating the principal component.

Extension to k dimensions

To reduce from n dimensions to k dimensions, select the top k eigenvectors of Σ corresponding to the k largest eigenvalues, creating a new k-dimensional representation y_i = [u_1^T x_i, ..., u_k^T x_i].

Mandatory preprocessing requirements

Data must be standardized to zero mean and unit variance before applying PCA to ensure the algorithm captures correlations rather than being dominated by features with larger scales.

Lossy compression and reconstruction

The original high-dimensional data can be approximately reconstructed from the reduced representation using x_i ≈ Σ_{j=1}^k y_{ij} u_j, projecting back onto the principal component subspace.

Appropriate scale for dimensionality reduction

PCA is designed for high-dimensional data reduction such as 10,000 dimensions to 100 dimensions, not trivial reductions from 2D to 1D as commonly illustrated in tutorials.

Warning against routine application

Ng explicitly cautions that PCA is frequently misapplied when practitioners should work with original features instead, emphasizing that it should only be used when dimensionality reduction specifically serves the downstream task.