Cross-Entropy Loss

machine-learning statistics optimisation

Definition

Cross-Entropy Loss

Cross-entropy loss is a loss function for probabilistic classification models. It measures how much probability the model assigns to the labels that actually occur.

Let $P$ be the target probability distribution over labels and let $Q$ be the model’s predicted distribution. The cross-entropy from $P$ to $Q$ is

$$H(P,Q)=-{\mathbb{E}}_{Y\sim P}[\log Q(Y)]=-\sum _{y}P(y)\log Q(y).$$

For a single training example with true label $y$, this becomes the negative log-probability assigned to the correct class:

$$\ell (y,Q)=-\log Q(y).$$

Thus cross-entropy loss is small when the model assigns high probability to the correct label, and large when it assigns low probability to the correct label.

Interpretation

Cross-entropy is the expected surprise under the true distribution when outcomes are encoded using the model distribution. If the model says the observed label was likely, the loss is small. If the model says the observed label was unlikely, the loss is large.

This makes it a natural training objective for probabilistic classifiers: the model is not only asked to choose the right class, but to place probability mass on it.

Relation to KL Divergence

Cross-entropy decomposes as

$$H(P,Q)=H(P)+D_{\mathrm{KL}}(P\Vert Q),$$

where $H(P)$ is the entropy of the target distribution and $D_{\mathrm{KL}}(P\Vert Q)$ is the Kullback–Leibler divergence from $P$ to $Q$.

When $P$ is fixed, minimising cross-entropy in $Q$ is therefore equivalent to minimising $D_{\mathrm{KL}}(P\Vert Q)$.

Binary Case

For binary classification, cross-entropy becomes binary cross-entropy loss. With true label $y\in \{0,1\}$ and predicted conditional probability $q=P(Y=1\mid X=x)$, it is

$$\ell (y,q)=-(y\log q+(1-y)\log (1-q)).$$

This is the standard loss for logistic regression and binary neural classifiers with a sigmoid output.

Maximum Likelihood

Minimising cross-entropy over a dataset is equivalent to maximising the likelihood of the observed labels under the model’s predicted distributions.

For independent examples $(x_i,y_i)$, minimising

$$-\sum _i\log Q_i(y_i)$$

maximises the probability that the model assigns to the observed label sequence.