Boosting

machine-learning ensemble-learning

Definition

Boosting

Boosting is an iterative ensemble learning technique that aims to reduce the bias of a model by combining multiple “weak” learners into a single “strong” learner. Formally, boosting trains base learners $\{h_1,h_2,\dots ,h_k\}$ sequentially, where each subsequent learner focuses on correcting the errors (residuals) of its predecessors.

Gradient Boosting

In gradient boosting, each new model $h_j$ is trained to minimise the residual risk of the current ensemble $h^{\ast }$. Given a learning rate $\eta$, the ensemble is updated as:

$$h_{new}^{\ast }=h_{old}^{\ast }+\eta \cdot h_j$$

where $h_j\approx \arg \min \sum (r_i-h(x_i){)}^2$ and $r_i$ are the residual errors $y_i-h_{old}^{\ast }(x_i)$. This procedure can be viewed as performing gradient descent in the function space.

Line Search: In practical implementations, the fixed learning rate $\eta$ is often replaced by an adaptive step size ${\eta }_j$ identified via line search. This involves solving a one-dimensional optimisation problem at each step to find the ${\eta }_j$ that minimises the loss of the updated ensemble: ${\eta }_j=\arg {\min }_{{\eta }^{\prime }}\mathcal{L}(h_{old}^{\ast }+{\eta }^{\prime }{h}_j)$.