Quadratic Loss

machine-learning optimisation

Definition

Quadratic Loss

Quadratic loss is the loss function that penalises a prediction by the square of its residual. For a numerical target $y\in \mathbb{R}$ and prediction $\hat{y}\in \mathbb{R}$, let

$$r=\hat{y}-y.$$

The quadratic loss is

$$\ell (\hat{y},y)=\frac{1}{2}(\hat{y}-y{)}^2=\frac{1}{2}{r}^2.$$

The factor $1/2$ does not change the minimiser; it cancels the $2$ when differentiating. The loss is small near the correct prediction and grows quadratically as the residual moves away from zero, so large errors are penalised disproportionately.

From residuals to a quadratic surface

For a dataset $S=\{(x_i,y_i){\}}_{i=1}^m$, the average quadratic loss is the mean squared error up to the factor $1/2$:

$$L(w)=\frac{1}{2m}\sum _{i=1}^m(h_w(x_i)-y_i{)}^2.$$

If the model is linear, $h_w(x)=w^{\top }x$, then $L(w)$ is a quadratic function of the parameters:

$$L(w)=\frac{1}{2}{w}^{\top }Hw-b^{\top }w+c,$$

where $H$ is positive semidefinite. This is why squared-error regression has a bowl-shaped optimisation surface: the residuals are linear in $w$, and squaring them turns the objective into a quadratic surface.

Gradient and curvature

In one dimension, write

$$E(w)=\frac{1}{2}a{w}^2+bw+c,a>0.$$

Then

$$E^{\prime }(w)=aw+b,E^{\prime \prime }(w)=a.$$

The minimiser is the point where the derivative vanishes:

$$w^{\star }=-\frac{b}{a}.$$

Because the second derivative is constant, the local quadratic approximation is not merely an approximation. It is the whole objective.

Relation to the learning rate

For gradient descent on the one-dimensional quadratic,

$$w_{k+1}=w_k-\eta E^{\prime }(w_k),$$

the error $e_k=w_k-w^{\star }$ evolves as

$$e_{k+1}=(1-\eta a)e_k.$$

Thus the optimal fixed learning rate in this one-dimensional case is

$${\eta }^{\star }=\frac{1}{a}=\frac{1}{E^{\prime \prime }(w)}.$$

With this choice, gradient descent reaches the minimiser in one step. If $0<\eta a<1$, convergence is monotone; if $1<\eta a<2$, convergence oscillates; if $\eta a>2$, the iterates diverge.

In several dimensions, the same statement applies separately along the eigenvalue directions of the Hessian matrix. A single scalar learning rate must compromise between directions of small and large curvature.