Lagrange Multipliers

mathematics analysis optimisation

Definition

Lagrange Multipliers

The method of Lagrange multipliers is a strategy for finding the local maxima and minima of a function subject to equality constraints. Formally, to identify the extrema of an objective function $f(\mathbf{x})$ subject to $g(\mathbf{x})=0$, one constructs the Lagrangian $\mathcal{L}$:

$\mathcal{L}(\mathbf{x},\lambda )=f(\mathbf{x})-\lambda g(\mathbf{x})$

where $\lambda$ is the Lagrange multiplier. The solution is found by identifying the stationary points where the gradient of the Lagrangian vanishes: ${\nabla }_{\mathbf{x},\lambda }\mathcal{L}=0$.

Applications in Machine Learning

Principal Component Analysis: Used to identify the direction $\mathbf{w}$ that maximises variance subject to the unit-norm constraint $\Vert \mathbf{w}{\Vert }^2=1$. The resulting Lagrangian leads directly to the eigenvalue equation $C\mathbf{w}=\lambda \mathbf{w}$.

Support Vector Machines: Fundamental to the derivation of the optimal hyperplane. The primal SVM objective is converted into its dual form using Lagrange multipliers, enabling the use of the kernel trick.