Bias-Variance Tradeoff

machine-learning statistics

Definition

Bias-Variance Tradeoff

The bias-variance tradeoff is a fundamental property of statistical models where the expected prediction error is decomposed into the sum of the squared bias, the variance, and the irreducible noise. Formally, for a target function $y=f(x)+ϵ$, the expected squared error of a learner $h$ is:

$\mathbb{E}[(y-h(x){)}^2]=\text{Bias}[h(x){]}^2+\text{Var}[h(x)]+{\sigma }^2$

…

\draw[cyan!30!white, thick] (0.2, 2.2) .. controls (1.75, 0.4) .. (3.3, 1.8);
\node[text=cyan!30!white, font=\tiny\bfseries, anchor=west] at (3.3, 1.8) {risk};

\draw[dashed, cyan!30!white, thin] (1.75, 0.1) — (1.75, 0.85);
\node[text=cyan!30!white, font=\tiny\bfseries, anchor=south] at (1.75, 0.85) {best model};

% Axes
\draw[→, cyan!30!white] (0,0) — (3.8, 0) node[below, font=\tiny\bfseries, text=cyan!30!white] {model complexity};
\draw[→, cyan!30!white] (0,0) — (0, 2.5);
\end{tikzpicture}
\end{document}

Error Components

Bias (Approximation Error): $\text{Bias}[h(x)]=\mathbb{E}[h(x)]-f(x)$. High bias indicates that the model’s assumptions are too restrictive to capture the underlying data structure, leading to underfitting.

Variance (Estimation Error): $\text{Var}[h(x)]=\mathbb{E}[h(x{)}^2]-\mathbb{E}[h(x){]}^2$. High variance indicates that the model is overly sensitive to small fluctuations in the training data, leading to overfitting.