Regularisation

machine-learning optimisation

Definition

Regularisation

Regularisation is a technique used to prevent overfitting by penalising model complexity within the objective function. The objective is to identify a hypothesis $h$ that achieves a low empirical risk while maintaining the simplest possible structure. Formally, a regularisation term $\Omega [h]$ is added to the loss function, resulting in the structural risk:

$\mathcal{L}(h,(x,y))=L(h(x),y)+\lambda \Omega [h]$

where $\lambda \ge 0$ is the regularisation parameter that controls the tradeoff between fitting the training data and keeping the model parameters small.

Penalisation Schemes

The choice of $\Omega [h]$ determines the geometric properties of the parameter space and the resulting model characteristics.

L1 Regularisation (Lasso): Utilises the $L_1$ norm of the weights, $\Omega [h]=\sum |w_i|$, which promotes sparsity and performs automatic feature selection.

L2 Regularisation (Ridge): Utilises the squared $L_2$ norm, $\Omega [h]=\sum w_i^2$, which penalises large weights more heavily and improves model stability and generalisation.