Coefficient of Determination

machine-learning statistics

Definition

Coefficient of Determination

Let $y_1,\dots ,y_N\in \mathbb{R}$ be observed values with mean $\bar{y}$, and let ${\hat{y}}_1,\dots ,{\hat{y}}_N$ be the corresponding predictions from a regression model. The coefficient of determination $R^2$ is:

$$R^2=1-\frac{{\sum }_{i=1}^N(y_i-{\hat{y}}_i{)}^2}{{\sum }_{i=1}^N(y_i-\bar{y}{)}^2}$$

The numerator is the sum of squared residuals (unexplained variance). The denominator is the total sum of squares (total variance).

Interpretation

$R^2$ measures the proportion of variance in the dependent variable that the model explains.

Value	Meaning
$R^2=1$	Perfect fit; all residuals are zero.
$R^2=0$	The model predicts no better than the mean $\hat{y}$.
$R^2<0$	The model is worse than predicting the mean; the numerator exceeds the denominator.

Properties

Not a measure of correctness

A high $R^2$ does not imply that the model is correct, only that it fits the observed data well. A misspecified model can still achieve $R^2\approx 1$ if it has enough parameters or if the data happen to align.

Sensitive to outliers

Because $R^2$ is based on squared residuals, a single large outlier can substantially reduce its value.

Increases with added predictors

Adding any predictor to a linear regression never decreases $R^2$, even if the predictor is irrelevant. The adjusted $R^2$ penalises extra parameters to compensate.