Kullback–Leibler Divergence

statistics information-theory probability

Definition

Kullback–Leibler Divergence

The Kullback–Leibler divergence is a divergence function that measures how much a probability distribution $Q$ differs from a reference distribution $P$.

For discrete distributions $P$ and $Q$ on the same set $\mathcal{X}$, with probability mass functions $p$ and $q$, it is defined by

$$D_{\mathrm{KL}}(P\Vert Q)=\sum _{x\in \mathcal{X}}p(x)\log \frac{p(x)}{q(x)}.$$

The first argument $P$ is the distribution being represented, and the second argument $Q$ is the approximation. If $q(x)=0$ for some $x$ with $p(x)>0$, then $D_{\mathrm{KL}}(P\Vert Q)=\infty$.

Continuous Case

For continuous distributions with densities $p$ and $q$, the sum is replaced by an integral:

$$D_{\mathrm{KL}}(P\Vert Q)=\int p(x)\log \frac{p(x)}{q(x)}dx.$$

This formula is valid when $P$ is absolutely continuous with respect to $Q$, meaning $Q$ does not assign zero probability where $P$ assigns positive probability.

Interpretation

KL divergence measures the expected extra information cost of using $Q$ as a model when the true distribution is $P$.

If $Q$ assigns low probability to outcomes that are common under $P$, the ratio $\frac{p(x)}{q(x)}$ becomes large, and the divergence increases. If $P=Q$, every ratio is $1$, every logarithm is $0$, and the divergence is $0$.

Examples

Binary classification

In binary classification, the true conditional label distribution and the model’s predicted label distribution can both be represented as Bernoulli distributions.

Let

$$P(Y=1\mid X=x)=p,Q(Y=1\mid X=x)=q.$$

Then

$$P(Y=0\mid X=x)=1-p,Q(Y=0\mid X=x)=1-q.$$

The KL divergence from the true distribution $P$ to the model distribution $Q$ is

$$D_{\mathrm{KL}}(P\Vert Q)=p\log \frac{p}{q}+(1-p)\log \frac{1-p}{1-q}.$$

If the observed label is hard, $y\in \{0,1\}$, then the target distribution is concentrated on the observed class. In that case, minimising this KL divergence is equivalent to minimising the binary cross-entropy loss

$$-y\log q-(1-y)\log (1-q).$$

This is the probabilistic view behind logistic regression: the model predicts $q$, and training penalises it when it assigns low probability to the observed class.