Student-t Test

statistics evaluation

Definition

Student-t Test

The Student-t test is a statistical hypothesis test utilised to determine if there is a significant difference between the means of two groups.

In the context of machine learning, it is employed to verify if the observed difference in performance metrics (e.g., accuracy) between two models is statistically significant or merely due to random chance.

Operational Framework

P-value Selection: Before conducting the test, a significance level (typically ) is selected. The test produces a p-value; if , the null hypothesis (that there is no difference between the models) is rejected.

Application in Evaluation: When comparing models using k-fold cross-validation, the t-test evaluates the means of the metrics obtained across the folds. This ensures that a “best” model is selected based on statistically robust evidence rather than variance in the specific data splits.