Biased Estimator

probability-theory

Definition

Biased Estimator

Let $\theta$ be a parameter and $\hat{\theta }=T(X_1,\dots ,X_n)$ an estimator built from a random sample. The estimator is biased for $\theta$ when its bias is nonzero:

$$\mathrm{Bias}[\hat{\theta }]=\mathbb{E}[\hat{\theta }]-\theta \ne 0.$$

A positive bias means $\hat{\theta }$ systematically overestimates $\theta$; a negative bias means it systematically underestimates. The deviation is a property of the estimator’s construction, not of any particular sample, so it persists as the sample grows.

Bias is not always disqualifying

A biased estimator can still beat an unbiased one. By the mean squared error decomposition

$$\mathrm{MSE}[\hat{\theta }]=\mathrm{Bias}[\hat{\theta }{]}^2+\mathrm{Var}(\hat{\theta }),$$

trading a small amount of bias for a large reduction in variance lowers the total error. This is the principle behind shrinkage and regularisation, formalised by the bias-variance tradeoff.