Bias

statistics

Definition

Bias

The bias of an estimator $\hat{\theta }$ for a quantity $\theta$ is the difference between its expected value and the true value:

$$\mathrm{Bias}[\hat{\theta }]=\mathbb{E}[\hat{\theta }]-\theta .$$

An unbiased estimator has $\mathrm{Bias}[\hat{\theta }]=0$, i.e. it is correct on average over repeated samples. Bias is a property of the estimator, not of any single sample: it measures systematic deviation, the error that survives even with infinitely many samples.

Bias in the mean squared error

Bias is one of two independent sources of error in an estimator. The mean squared error decomposes as

$$\mathrm{MSE}[\hat{\theta }]=\mathbb{E}[(\hat{\theta }-\theta {)}^2]=\mathrm{Bias}[\hat{\theta }{]}^2+\mathrm{Var}(\hat{\theta }),$$

so a biased estimator can still be preferable when its variance is low enough that the total error shrinks. Balancing the two terms is the bias-variance tradeoff. In supervised learning the same idea appears at each input $x$ as $\mathbb{E}[h(x)]-f(x)$, the bias of a predictor, whose learning-theory counterpart is the approximation error.