Definition
Bias-Variance Tradeoff
The Bias-Variance Tradeoff is a central dilemma in supervised learning that describes the inverse relationship between two sources of error that prevent models from generalising perfectly to unseen data: bias and variance. The challenge is that decreasing one type of error often increases the other, and the goal is to find a model complexity that achieves the best balance to minimise total error.
Bias: This is the error introduced by approximating a real-world problem, which may be complex, with a model that is too simple. High-bias models make strong assumptions about the data (e.g., assuming a linear relationship) and are unable to capture its underlying structure, leading to underfitting. They consistently miss the mark, regardless of the training data.
The model makes wrong assumptions about the data set.Variance: This is the error introduced because a model is too sensitive to the specific training data it was given. A high-variance model learns not only the underlying signal but also the random noise in the training data. This leads to overfitting. The model performs exceptionally well on the training data but poorly on unseen data because it changes drastically with small fluctuations in the training set.
The model can adjust based on the data set.