First-order Optimisation Method

optimisation calculus machine-learning

Definition

First-order Optimisation Method

A first-order optimisation method is an optimisation method that chooses its updates using first-derivative information of an objective, usually the gradient, but not second-derivative curvature information such as the Hessian matrix. For a differentiable objective $f:{\mathbb{R}}^d\to \mathbb{R}$, a typical first-order step has the form

$${\theta }_{t+1}={\theta }_t-{\eta }_t{d}_t,$$

where $d_t$ is computed from $\nabla f({\theta }_t)$, and ${\eta }_t>0$ is a learning rate. Gradient descent is the canonical example, with $d_t=\nabla f({\theta }_t)$. The method reads the local slope of the objective, but it does not directly rescale directions by local curvature as a second-order optimisation method does.