Definition
Ensemble Learning Algorithm
An ensemble learning algorithm is a meta-algorithm that constructs a set of base hypotheses and combines their outputs to produce a single aggregated prediction. Formally, given a training dataset , the ensemble learner generates a composite hypothesis , where is an aggregation function such as weighted averaging or majority voting.
Error Reduction Strategies
Ensemble methods are typically employed to improve generalisation by balancing the components of the bias-variance tradeoff.
Variance Reduction (Bagging): Parallel ensembles, such as Random Forests, train base learners independently on different subsets of the data to reduce model variance.
Bias Reduction (Boosting): Sequential ensembles, such as AdaBoost or Gradient Boosting, train learners iteratively, where each model focuses on correcting the specific errors of its predecessor.