Empirical Risk

machine-learning statistics

Definition

Empirical Risk

Empirical risk, denoted $R_{\text{emp}}(h)$, is the expected loss of a hypothesis $h$ over a finite training dataset $D=\{(x_i,y_i){\}}_{i=1}^n$. It serves as a computable proxy for the theoretical true risk. Formally:

$$R_{\text{emp}}(h)=\frac{1}{n}\sum _{i=1}^{n}L(h(x_i),y_i)$$

where $L$ is the chosen loss function.

Risk Minimisation

The fundamental principle of Empirical Risk Minimisation (ERM) involves selecting a hypothesis $\hat{h}$ from the hypothesis class $\mathcal{H}$ that minimises the error on the observed sample:
$\hat{h}=\arg {\min }_{h\in \mathcal{H}}{R}_{\text{emp}}(h)$
While ERM is computationally feasible, relying solely on empirical risk without regularisation can lead to overfitting, where the model captures sample-specific noise rather than the underlying distribution.