Risk

machine-learning

Definition

Risk

Risk, also known as the generalisation error or true risk, is the expected loss of a model over the true, underlying data distribution. It quantifies how well a model is expected to perform on unseen data points drawn from the same distribution as the training data.

Let $P(x,y)$ be the true joint probability distribution over the input-output space $X\times Y$. Let $h:X\to Y$ be our model (hypothesis) and $L(h(x),y)$ be the loss function that measures the cost of predicting $h(x)$ when the true label is $y$.

The risk of the hypothesis $h$, denoted as $R(h)$, is the expected value of the loss with respect to the true data distribution $P$:

$$R(h)={\mathbb{E}}_{(x,y)\sim P}[L(h(x),y)]$$

This can be expressed as an integral (for continuous variables) or a sum (for discrete variables) over all possible data points, weighted by their probability of occurrence.

True Risk vs. Empirical Risk

It is crucial to distinguish between true risk and empirical risk:

True Risk:

Definition

True Risk

True risk $R(h)$, $h$ being a model is the expected loss on all possible data, including future, unseen examples. In practice, $R(h)$ is incomputable because the true distribution $P(x,y)$ is unknown.

Link to original

Empirical Risk:

Definition

Empirical Risk

Empirical Risk $R_{emp}(h)$, $h$ being a model, is the average loss calculated on a finite training dataset $S=\{(x_1,y_1),\dots ,(x_n,y_n)\}$. It serves as a practical, computable estimate of the true risk.

$$R_{emp}(h)=\frac{1}{n}\sum _{i=1}^{n}L(h(x_i),y_i)$$

Link to original

The ultimate goal of a machine learning algorithm is to find a model $h$ that minimises the true risk $R(h)$. However, since we only have access to the training data, we minimise the empirical risk $R_{emp}(h)$ as a proxy. A significant gap between low empirical risk and high true risk is known as overfitting.