Second-order Optimisation Method

optimisation calculus machine-learning

Definition

Second-order Optimisation Method

A second-order optimisation method is an optimisation method that uses both the gradient and second-derivative curvature information of an objective. For a twice differentiable function $f:{\mathbb{R}}^d\to \mathbb{R}$, this curvature is encoded by the Hessian matrix

$$H_t={\nabla }^{2}f({\theta }_t).$$

A typical second-order update has the form

$${\theta }_{t+1}={\theta }_t-{\nu }_t{B}_t^{-1}\nabla f({\theta }_t),$$

where $B_t$ is the Hessian or an approximation to it, and ${\nu }_t>0$ is a damping or step-size parameter. The gradient gives the local downhill direction; the Hessian rescales that direction according to local curvature.