A Large Learning-Rate Ratio Makes Descent Slow

A neural network loss landscape is not usually shaped like a round bowl. Near a point, some directions can be sharp and others can be flat. A learning rate is only one scalar, so it has to move through all of those directions with the same global scale.

The difficulty can be measured by the spread of the direction-wise optimal learning rates:

$$\frac{{\max }_i{\eta }_{i,\mathrm{opt}}}{{\min }_i{\eta }_{i,\mathrm{opt}}}.$$

When this ratio is large, convergence is slow because the directions disagree about what a good step size is.

The left end of the axis is controlled by sharp directions. The right end is controlled by flat directions. A first-order method must choose one scalar $\eta$, so it usually chooses something safe for the sharp directions. That makes the flat directions painfully slow.

Each direction has its own ideal step

Near a well-behaved point, a smooth loss can be approximated by a quadratic in the eigenbasis of the Hessian matrix:

$$L({\theta }^{\star }+z)\approx L({\theta }^{\star })+\frac{1}{2}\sum _i{\lambda }_i{z}_i^2.$$

Here the eigenvalue ${\lambda }_i$ is the local curvature in direction $i$. In that one direction, gradient descent behaves like

$$z_{t+1,i}=(1-\eta {\lambda }_i)z_{t,i}.$$

The step size that would solve this one-dimensional quadratic in one step is

$${\eta }_{i,\mathrm{opt}}=\frac{1}{{\lambda }_i}.$$

So sharp directions, with large ${\lambda }_i$, have small optimal learning rates. Flat directions, with small ${\lambda }_i$, have large optimal learning rates.

A scalar step must obey the sharpest direction

For stability in every positive-curvature direction, the scalar learning rate must respect the largest curvature:

$$\eta <\frac{2}{{\lambda }_{\max }}.$$

But the flattest direction wants a step on the scale of

$$\frac{1}{{\lambda }_{\min }}.$$

The disagreement is exactly the spread of direction-wise ideal rates:

$$\frac{{\max }_i{\eta }_{i,\mathrm{opt}}}{{\min }_i{\eta }_{i,\mathrm{opt}}}=\frac{1/{\lambda }_{\min }}{1/{\lambda }_{\max }}=\frac{{\lambda }_{\max }}{{\lambda }_{\min }}.$$

When this number is large, the safe scalar step is set by the sharpest direction, while progress in the flattest direction is controlled by a multiplier close to one:

$$1-\eta {\lambda }_{\min }\approx 1.$$

That means the error in the flat direction shrinks only a little at each iteration.

Why this matters for neural networks

In neural networks, the Hessian spectrum can be very wide. Some parameter directions change the loss sharply; others barely change it. A first-order optimisation method sees the gradient, but its scalar learning rate cannot separately tune each curvature direction.

This is why training can feel slow even when every step is stable. The learning rate is not simply “too small”. It is small because some direction would become unstable if it were larger. The optimiser is moving with the speed limit imposed by the sharpest curvature while trying to make progress through much flatter directions.

A second-order optimisation method addresses this mismatch directly by using curvature information to rescale directions differently. Plain scalar learning-rate methods do not have that geometry built in.

A large ratio of optimal directional learning rates is therefore a warning that the landscape is badly scaled: one clock is being asked to keep time for many directions.