Kullback–Leibler Divergence

statistics information-theory probability

Definition

Kullback–Leibler Divergence

The Kullback–Leibler divergence is a divergence function that measures how much a probability distribution differs from a reference distribution .

For discrete distributions and on the same set , with probability mass functions and , it is defined by

The first argument is the distribution being represented, and the second argument is the approximation. If for some with , then .

Continuous Case

For continuous distributions with densities and , the sum is replaced by an integral:

This formula is valid when is absolutely continuous with respect to , meaning does not assign zero probability where assigns positive probability.

Interpretation

KL divergence measures the expected extra information cost of using as a model when the true distribution is .

If assigns low probability to outcomes that are common under , the ratio becomes large, and the divergence increases. If , every ratio is , every logarithm is , and the divergence is .

Examples

Binary classification

In binary classification, the true conditional label distribution and the model’s predicted label distribution can both be represented as Bernoulli distributions.

Let

Then

The KL divergence from the true distribution to the model distribution is

If the observed label is hard, , then the target distribution is concentrated on the observed class. In that case, minimising this KL divergence is equivalent to minimising the binary cross-entropy loss

This is the probabilistic view behind logistic regression: the model predicts , and training penalises it when it assigns low probability to the observed class.