L1 Regularisation

machine-learning optimisation

Definition

L1 Regularisation

L1 regularisation (or Lasso) is a regularisation technique that adds a penalty term proportional to the $L_1$ norm of the parameter vector to the loss function. Formally, the regularised loss is:

${\mathcal{L}}_{\text{reg}}=\mathcal{L}(h,(x,y))+\lambda {\sum }_{i=1}^m|w_i|$

where $w_i$ are the model weights and $\lambda \ge 0$ is the regularisation parameter.

Sparse Representations

Due to the geometric properties of the $L_1$ ball, which has vertices on the coordinate axes, L1 regularisation has the effect of driving the weights of less relevant features to exactly zero. This produces sparse models and effectively performs automatic feature selection, which is particularly beneficial in high-dimensional settings with many redundant features.