Principal Component Analysis

machine-learning linear-algebra statistics dimensionality-reduction

Definition

Principal Component Analysis

Principal Component Analysis (PCA) is a deterministic linear dimensionality reduction technique that identifies a lower-dimensional subspace that captures the maximum variance of the data. Formally, for a centred data matrix $X\in {\mathbb{R}}^{n\times p}$ with covariance matrix $C=\frac{1}{n-1}{X}^{\top }X$, PCA seeks a direction $\mathbf{w}$ with $\Vert \mathbf{w}\Vert =1$ that maximises the projected variance:

$\arg {\max }_{\Vert \mathbf{w}\Vert =1}\text{Var}(X\mathbf{w})=\arg {\max }_{\Vert \mathbf{w}\Vert =1}{\mathbf{w}}^{\top }C\mathbf{w}$

The solution is the eigenvector corresponding to the largest eigenvalue of the covariance matrix $C$.

Optimisation: The Lagrangian Approach

The objective is to maximise $L={\mathbf{w}}^{\top }C\mathbf{w}$ subject to the constraint ${\mathbf{w}}^{\top }\mathbf{w}=1$. Utilising the method of Lagrange multipliers:

$$\mathcal{L}(\mathbf{w},\lambda )={\mathbf{w}}^{\top }C\mathbf{w}-\lambda ({\mathbf{w}}^{\top }\mathbf{w}-1)$$

Taking the derivative with respect to $\mathbf{w}$ and equating to zero yields:
$\frac{\partial \mathcal{L}}{\partial \mathbf{w}}=2C\mathbf{w}-2\lambda \mathbf{w}=0⟹C\mathbf{w}=\lambda \mathbf{w}$
Substituting this back into the variance term gives ${\mathbf{w}}^{\top }C\mathbf{w}={\mathbf{w}}^{\top }\lambda \mathbf{w}=\lambda$. Thus, the variance is maximised by selecting $\mathbf{w}$ as the eigenvector associated with the largest eigenvalue ${\lambda }_{\text{max}}$.

Structural Properties

Orthogonality: The principal components (the projected directions) are mutually orthogonal and form a new basis for the feature space.

Reconstruction Error: Maximising the variance is mathematically equivalent to minimising the mean squared reconstruction error between the original data and its projection.

Linearity: As a linear method, PCA identifies subspaces (planes or hyperplanes). It may fail to capture non-linear structures, for which t-SNE or spectral clustering are more appropriate.

Numerical Stability: The quality of the principal components depends on the distribution of the eigenvalues. If the eigenvalues are roughly equal (${\lambda }_1\approx {\lambda }_2\approx \dots$), no single direction dominates the variance. In such cases, the principal components become unstable, as many different orthogonal directions capture approximately the same amount of information, making the model highly sensitive to small perturbations.