Bagging

machine-learning ensemble-learning

Definition

Bagging

Bagging (Bootstrap Aggregating) is an ensemble learning technique designed to reduce the variance of a model. Formally, given a dataset $D$ of size $m$, bagging generates $k$ new datasets $D_i$ by sampling $m$ instances from $D$ uniformly and with replacement (bootstrapping).

A separate model $h_i$ is trained on each $D_i$, and the final prediction is aggregated (e.g., majority voting for classification or averaging for regression):

$$H(\mathbf{x})=\frac{1}{k}\sum _{i=1}^k{h}_i(\mathbf{x})$$

Random Forests

A Random Forest is a specific application of bagging using decision trees as base learners, with the additional constraint that only a random subset of features is considered at each split, further decorrelating the trees.