Cross-Validation

machine-learning statistics

Definition

k-fold Cross-Validation

k-fold cross-validation is a robust evaluation technique used to estimate the true risk of a model, particularly when the dataset is small. Formally, the dataset $D$ is partitioned into $k$ equally sized, disjoint folds $\{F_1,F_2,\dots ,F_k\}$. The procedure involves $k$ iterations where:

1. Fold $F_i$ is held out as the test set.
2. The remaining $k-1$ folds are used as the training set.
3. A performance metric $M_i$ is computed on $F_i$.

The final performance estimate is the arithmetic mean of the metrics: $\bar{M}=\frac{1}{k}{\sum }_{i=1}^k{M}_i$.

Cross-Validation Variants

k-fold CV: The dataset is partitioned into $k$ equally sized folds. For each fold, the model is trained on the remaining $k-1$ folds and evaluated on the current fold. The overall metric is the average of these $k$ iterations.

Leave-One-Out (LOO): A special case of k-fold cross-validation where $k=n$ (the total number of samples). In each iteration, only a single observation is held out for testing. While providing an unbiased performance estimate, it is computationally expensive for large datasets as it requires training $n$ separate models.

Stratification

Stratified k-fold

In stratified k-fold cross-validation, the folds are constructed such that each fold contains approximately the same percentage of samples of each target class as the complete set, ensuring that each split is representative of the underlying class distribution.