Learning Rate

machine-learning optimisation

Definition

Learning Rate

The learning rate is a positive scalar hyperparameter that sets the step size of an iterative optimisation algorithm. In gradient descent, for an objective and parameters , it appears as

where scales the negative gradient direction. A small learning rate trusts the local slope cautiously; a large learning rate moves farther, but can overshoot the region where the local slope is reliable.

Role in gradient descent

The gradient gives a direction, not a distance. The learning rate supplies the distance. It decides how much of the local first-order model is used before the algorithm asks for a new gradient.

For a loss function in machine learning, this means the learning rate controls how strongly the model parameters react to the current training signal. Too small a value gives slow progress. Too large a value can make the parameters oscillate or diverge.

Quadratic stability thresholds

The thresholds for monotone convergence, oscillating convergence, and divergence hold exactly for gradient descent on a one-dimensional quadratic function, and they hold per eigenvalue direction for a multidimensional quadratic.

Let

Then

so the error evolves as

Everything is controlled by the multiplier :

learning-rate range	behaviour
	monotone convergence; the iterates stay on the same side of the optimum
	one-step convergence in this direction
	oscillating convergence; the iterates cross the optimum each step, but the error shrinks
	no convergence; the error flips sign with the same size
	divergence

For a quadratic objective

with positive definite Hessian matrix , the same analysis applies along each eigenvector of . If the eigenvalues are , then direction has multiplier .

Thus, writing :

gives convergence for every direction, while

prevents oscillation in every direction. If , the direction of largest curvature diverges.

Optimal fixed step size

For a one-dimensional quadratic, the best fixed step is

because it sends the error to zero in one step.

For a multidimensional strongly convex quadratic whose Hessian eigenvalues lie in , no single learning rate can be perfect for every direction unless . The best fixed learning rate in the worst-case spectral sense is

with contraction factor

This explains why ill-conditioned objectives are slow for plain gradient descent: when is large, one step size must serve both steep and shallow directions.

Beyond quadratics

For a general differentiable objective, the quadratic thresholds are local rules of thumb. Near a point, the Hessian describes local curvature, so the same idea applies approximately with local eigenvalues.

For an -smooth objective, the standard safe range is still governed by the largest curvature bound:

This guarantees descent under the usual smoothness assumptions, but it does not guarantee that the iterates move monotonically in parameter space. On non-convex objectives, a stable learning rate can still lead to a local minimum, a saddle point, or a flat region.