Divergence Function

machine-learning statistics

Definition

Divergence Function

A divergence function is a function that measures how different two objects are, without necessarily satisfying all axioms of a metric.

For a set $X$, a divergence function is usually a map

$$D:X\times X\to [0,\infty ]$$

such that

$$D(x,y)\ge 0\text{and usually}D(x,x)=0.$$

The value $D(x,y)$ measures the discrepancy between $x$ and $y$. Unlike a metric, a divergence function may be asymmetric, may fail the triangle inequality, and may treat its two arguments differently.

Interpretation

A divergence is often used when one object is treated as the reference and the other as an approximation. In that case, $D(x,y)$ means: how costly is it to use $y$ in place of $x$?

This is why the order may matter. A Kullback–Leibler divergence $D_{\mathrm{KL}}(P\Vert Q)$ measures the cost of approximating $P$ by $Q$, and this is generally different from approximating $Q$ by $P$.

Comparison with Metrics

Every non-negative metric can be used as a divergence, but not every divergence is a metric.

Property	Metric	Divergence function
Non-negative	yes	usually yes
Zero on identical objects	yes	usually yes
Symmetric	required	not required
Triangle inequality	required	not required

Thus a divergence is a broader notion of discrepancy. It keeps the idea of separation, but drops some geometric constraints when they are not appropriate for the problem.

Examples

• L2 divergence measures squared Euclidean discrepancy between vectors.
• Kullback–Leibler divergence measures discrepancy between probability distributions.