Estimation Error

machine-learning learning-theory

Definition

Estimation Error

The estimation error is the gap between the empirical-risk minimiser $\hat{h}$ returned from a finite training set and the best model its hypothesis class $\mathcal{H}$ could express. Writing

$$h^{\star }=\arg \underset{h\in \mathcal{H}}{\min }R(h)$$

for the best-in-class, it is

$$R(\hat{h})-R(h^{\star }).$$

It is the cost of choosing $\hat{h}$ from one sample rather than knowing $h^{\star }$ exactly: different training sets yield different $\hat{h}$, and the spread of those picks is the learning-theory counterpart of variance. Together with the approximation error, it partitions the excess risk over the Bayes optimum (see the decomposition).

What drives it

Estimation error falls as the sample grows — the empirical risk concentrates toward the true risk, so $\hat{h}$ tends to $h^{\star }$. It rises with the capacity of $\mathcal{H}$: a richer class fits any given sample more closely, so $\hat{h}$ varies more across samples. When this term dominates, the model tracks the particular sample rather than the underlying pattern, the familiar symptom of overfitting.