L2 Divergence

linear-algebra statistics machine-learning

Definition

L2 Divergence

The L2 divergence is the divergence function that measures squared L2 norm discrepancy between two vectors.

For $x,y\in {\mathbb{R}}^d$, it is defined by

$$D_{L_2}(x,y)=\Vert x-y{\Vert }_2^2=\sum _{i=1}^d(x_i-y_i{)}^2.$$

Some conventions include a factor $\frac{1}{2}$, giving $\frac{1}{2}\Vert x-y{\Vert }_2^2$, because it simplifies gradients. Both forms measure the same discrepancy up to a constant factor.

Interpretation

L2 divergence compares two vectors coordinate by coordinate, squares each difference, and adds the squared errors. Large coordinate errors are penalised more strongly than small ones.

The unsquared quantity $\Vert x-y{\Vert }_2$ is the Euclidean distance. Squaring it gives a divergence that is smooth and convenient for optimisation, but it is no longer a metric because the triangle inequality need not hold.

Gradient

For the common half-scaled version

$$\frac{1}{2}\Vert x-y{\Vert }_2^2,$$

the gradient with respect to $y$ is

$${\nabla }_y\frac{1}{2}\Vert x-y{\Vert }_2^2=y-x.$$

This is why the factor $\frac{1}{2}$ is common in optimisation: it cancels the factor $2$ from differentiating the square.