Loss

machine-learning optimisation

Definition

Loss

A loss function assigns a non-negative cost to a prediction on one example. For an input space $\mathcal{X}$, a label space $\mathcal{Y}$, a prediction space $\mathcal{A}$, and a predictor $h:\mathcal{X}\to \mathcal{A}$, it is a function

$$\ell :\mathcal{A}\times \mathcal{Y}\to {\mathbb{R}}_{\ge 0},\ell (h(x),y),$$

where $\ell (h(x),y)$ measures how bad the prediction $h(x)$ is when the true label is $y$. A loss function evaluates one prediction; a risk averages that loss over a data distribution or a training sample.

Role in learning

In supervised learning, the loss turns prediction quality into an optimisation objective. A hypothesis $h\in \mathcal{H}$ is preferred when it has smaller expected loss:

$$R(h)={\mathbb{E}}_{(X,Y)\sim P}[\ell (h(X),Y)].$$

On a finite training set $S=\{(x_i,y_i){\}}_{i=1}^m$, this is approximated by the empirical average

$$R_S(h)=\frac{1}{m}\sum _{i=1}^m\ell (h(x_i),y_i).$$

Different losses ask different questions. Zero-one loss asks only whether the predicted label is wrong. Cross-entropy loss asks how much probability the model assigned to the true label. Quadratic loss asks how far a numerical prediction is from a numerical target, with large residuals punished more strongly.