Optimisation Method

optimisation calculus machine-learning

Definition

Optimisation Method

An optimisation method is a systematic procedure for searching for a point that minimises or maximises an objective function. For a minimisation problem

$$\underset{\theta \in \Theta }{\min }f(\theta ),$$

an optimisation method produces a sequence of candidate points

$${\theta }_0,{\theta }_1,{\theta }_2,\dots$$

using information about $f$, the feasible set $\Theta$, and possibly past iterates. The method specifies how each new point is chosen, when the search stops, and what kind of optimality it is expected to approach. First-order optimisation methods use gradient information; second-order optimisation methods also use curvature information.