Learning Rate

machine-learning optimisation

Definition

Learning Rate

The learning rate is a positive scalar hyperparameter that sets the step size of an iterative optimisation algorithm. In gradient descent, for an objective $f$ and parameters ${\theta }_t$, it appears as

$${\theta }_{t+1}={\theta }_t-{\eta }_t\nabla f({\theta }_t),$$

where ${\eta }_t>0$ scales the negative gradient direction. A small learning rate trusts the local slope cautiously; a large learning rate moves farther, but can overshoot the region where the local slope is reliable.

Role in gradient descent

The gradient gives a direction, not a distance. The learning rate supplies the distance. It decides how much of the local first-order model is used before the algorithm asks for a new gradient.

For a loss function in machine learning, this means the learning rate controls how strongly the model parameters react to the current training signal. Too small a value gives slow progress. Too large a value can make the parameters oscillate or diverge.

Quadratic stability thresholds

The thresholds for monotone convergence, oscillating convergence, and divergence hold exactly for gradient descent on a one-dimensional quadratic function, and they hold per eigenvalue direction for a multidimensional quadratic.

Let

$$f(\theta )=\frac{\lambda }{2}(\theta -{\theta }^{\star }{)}^2,\lambda >0.$$

Then

$$\nabla f({\theta }_t)=\lambda ({\theta }_t-{\theta }^{\star }),$$

so the error $e_t={\theta }_t-{\theta }^{\star }$ evolves as

$$e_{t+1}=(1-\eta \lambda )e_t.$$

Everything is controlled by the multiplier $1-\eta \lambda$:

learning-rate range	behaviour
$0<\eta \lambda <1$	monotone convergence; the iterates stay on the same side of the optimum
$\eta \lambda =1$	one-step convergence in this direction
$1<\eta \lambda <2$	oscillating convergence; the iterates cross the optimum each step, but the error shrinks
$\eta \lambda =2$	no convergence; the error flips sign with the same size
$\eta \lambda >2$	divergence

For a quadratic objective

$$f(\theta )=\frac{1}{2}{\theta }^{\top }H\theta -b^{\top }\theta ,$$

with positive definite Hessian matrix $H$, the same analysis applies along each eigenvector of $H$. If the eigenvalues are ${\lambda }_i$, then direction $i$ has multiplier $1-\eta {\lambda }_i$.

Thus, writing $L={\lambda }_{\max }(H)$:

$$0<\eta <\frac{2}{L}$$

gives convergence for every direction, while

$$0<\eta \le \frac{1}{L}$$

prevents oscillation in every direction. If $\eta >2/L$, the direction of largest curvature diverges.

Optimal fixed step size

For a one-dimensional quadratic, the best fixed step is

$${\eta }^{\star }=\frac{1}{\lambda },$$

because it sends the error to zero in one step.

For a multidimensional strongly convex quadratic whose Hessian eigenvalues lie in $[\mu ,L]$, no single learning rate can be perfect for every direction unless $\mu =L$. The best fixed learning rate in the worst-case spectral sense is

$${\eta }^{\star }=\frac{2}{L+\mu },$$

with contraction factor

$$\rho =\frac{L-\mu }{L+\mu }.$$

This explains why ill-conditioned objectives are slow for plain gradient descent: when $L/\mu$ is large, one step size must serve both steep and shallow directions.

Beyond quadratics

For a general differentiable objective, the quadratic thresholds are local rules of thumb. Near a point, the Hessian describes local curvature, so the same idea applies approximately with local eigenvalues.

For an $L$-smooth objective, the standard safe range is still governed by the largest curvature bound:

$$0<\eta <\frac{2}{L}.$$

This guarantees descent under the usual smoothness assumptions, but it does not guarantee that the iterates move monotonically in parameter space. On non-convex objectives, a stable learning rate can still lead to a local minimum, a saddle point, or a flat region.