Hyperparameter

machine-learning optimisation

Definition

Hyperparameter

A hyperparameter is a configuration variable that is external to the model and whose value cannot be estimated from the data during the standard training process. Formally, while parameters $\theta$ are identified by minimising the empirical risk on a training dataset $D_{\text{train}}$, hyperparameters $\lambda$ are typically selected by evaluating model performance on a disjoint validation set $D_{\text{val}}$.

Examples include:

• The learning rate $\eta$ in gradient descent.
• The regularisation parameter $\lambda$ in regularisation.
• The maximum depth or stopping criteria in a decision tree.