Kullback–Leibler Divergence

statistics information-theory probability

Definition

Kullback–Leibler Divergence

The Kullback–Leibler (KL) divergence is a non-symmetric measure of the difference between two probability distributions $P$ and $Q$ defined over the same probability space. Formally, for discrete distributions, it is defined as:

$D_{KL}(P\Vert Q)={\sum }_{x\in \mathcal{X}}P(x)\ln \left(\frac{P(x)}{Q(x)}\right)$

For continuous distributions, the sum is replaced by an integral. It quantifies the information lost when $Q$ is used to approximate $P$.

Properties

Non-negativity: According to Gibbs’ inequality, $D_{KL}(P\Vert Q)\ge 0$ for all distributions, with equality holding if and only if $P=Q$.

Asymmetry: $D_{KL}(P\Vert Q)\ne D_{KL}(Q\Vert P)$. Consequently, KL divergence is not a formal metric, as it violates the symmetry and triangle inequality axioms.

Application in ML: It is the foundational objective for many algorithms, including t-SNE, where it is minimised to align low-dimensional embeddings with high-dimensional data structures, and in variational inference.