Definition
Hyperparameter
A hyperparameter is a configuration variable that is external to the model and whose value cannot be estimated from the data during the standard training process. Formally, while parameters
are identified by minimising the empirical risk on a training dataset , hyperparameters are typically selected by evaluating model performance on a disjoint validation set . Examples include:
- The learning rate
in gradient descent. - The regularisation parameter
in regularisation. - The maximum depth or stopping criteria in a decision tree.