Bias-Variance Tradeoff

machine-learning statistics

Definition

Bias-Variance Tradeoff

In supervised regression with target $y=f(x)+\epsilon$, $\mathbb{E}[\epsilon ]=0$ and $\mathrm{Var}(\epsilon )={\sigma }^2$, the expected squared error of a learner $h$ — averaged over training samples and noise — decomposes as

$\mathbb{E}[(y-h(x){)}^2]=\mathrm{Bias}[h(x){]}^2+\mathrm{Var}(h(x))+{\sigma }^2,$

from three independent sources:

• Bias $\mathrm{Bias}[h(x)]=\mathbb{E}[h(x)]-f(x)$: how far the average prediction misses $f$, the cost of a hypothesis class that cannot represent it.
• Variance $\mathrm{Var}(h(x))$: how much the prediction shifts between training sets drawn from the same distribution.
• Irreducible noise ${\sigma }^2$: the variance of $\epsilon$, a floor no model can beat.

The tradeoff is that the two model-dependent terms move against each other: enriching the class lowers bias by letting $f$ be represented, but raises variance by making $\hat{h}$ more sensitive to the particular sample.

…

\draw[cyan!30!white, thick] (0.2, 2.2) .. controls (1.75, 0.4) .. (3.3, 1.8);
\node[text=cyan!30!white, font=\tiny\bfseries, anchor=west] at (3.3, 1.8) {risk};

\draw[dashed, cyan!30!white, thin] (1.75, 0.1) — (1.75, 0.85);
\node[text=cyan!30!white, font=\tiny\bfseries, anchor=south] at (1.75, 0.85) {best model};

% Axes
\draw[→, cyan!30!white] (0,0) — (3.8, 0) node[below, font=\tiny\bfseries, text=cyan!30!white] {model complexity};
\draw[→, cyan!30!white] (0,0) — (0, 2.5);
\end{tikzpicture}
\end{document}

Bias

Definition

Approximation Error

The approximation error is the gap between the risk of the best model a hypothesis class $\mathcal{H}$ can express and the lowest risk attainable by any predictor. Writing

$$h^{\star }=\arg \underset{h\in \mathcal{H}}{\min }R(h)$$

for the best-in-class model and $R^{\star }$ for the Bayes risk, it is

$$R(h^{\star })-R^{\star }.$$

It measures a limitation of $\mathcal{H}$ itself, independent of any data: even with infinitely many samples, no model in $\mathcal{H}$ can beat $h^{\star }$. A class too restrictive for the target leaves a large approximation error, the learning-theory counterpart of bias and the source of underfitting. It is reduced only by enlarging or changing $\mathcal{H}$, never by collecting more data.

Link to original

Variance

Definition

Estimation Error

The estimation error is the gap between the empirical-risk minimiser $\hat{h}$ returned from a finite training set and the best model its hypothesis class $\mathcal{H}$ could express. Writing

$$h^{\star }=\arg \underset{h\in \mathcal{H}}{\min }R(h)$$

for the best-in-class, it is

$$R(\hat{h})-R(h^{\star }).$$

It is the cost of choosing $\hat{h}$ from one sample rather than knowing $h^{\star }$ exactly: different training sets yield different $\hat{h}$, and the spread of those picks is the learning-theory counterpart of variance. Together with the approximation error, it partitions the excess risk over the Bayes optimum (see the decomposition).

Link to original