Mean Squared Error

machine-learning statistics regression

Definition

Mean Squared Error

The mean squared error (MSE) is the average quadratic loss over a dataset. For targets $y_1,\dots ,y_N$ and predictions ${\hat{y}}_1,\dots ,{\hat{y}}_N$ with residuals $r_i=y_i-{\hat{y}}_i$,

$$\text{MSE}=\frac{1}{N}\sum _{i=1}^N(y_i-{\hat{y}}_i{)}^2=\frac{1}{N}\sum _{i=1}^N{r}_i^2.$$

Squaring makes every deviation non-negative and penalises large residuals disproportionately: an error of size $2c$ contributes $4c^2$, four times the contribution of an error of size $c$. The square root yields the RMSE, which restores the target’s units.

Decomposition

For an estimator $\hat{\theta }$ of a quantity $\theta$, the MSE decomposes into a bias term and a variance term:

$$\mathrm{MSE}(\hat{\theta },\theta )=\mathbb{E}\left[(\hat{\theta }-\theta {)}^2\right]=\underset{\text{bias2}}{\underbrace{\mathrm{Bias}(\hat{\theta },\theta {)}^2}}+\underset{\text{variance}}{\underbrace{\mathrm{Var}(\hat{\theta })}}.$$

A biased estimator can therefore attain a lower MSE than an unbiased one, by trades a small bias for a larger reduction in variance.