Gradient Descent

machine-learning optimisation calculus

Definition

Gradient Descent

Gradient descent is a first-order iterative optimisation algorithm used to identify a local minimum of a differentiable function. In the context of model training, it minimises a loss function $J(\theta )$ by updating parameters $\theta$ in the direction of the steepest descent.

The update rule for iteration $t$ is:
${\theta }_{t+1}={\theta }_t-\eta \cdot {\nabla }_{\theta }J({\theta }_t)$

Convergence Properties

Learning Rate Impact: The hyperparameter $\eta$ (step size) is critical; excessive values may lead to divergence, while insufficient values result in slow convergence.

Local Optima: For non-convex objective functions, the algorithm may converge to a local minimum or a saddle point rather than the global minimum.

Guarantees: Convergence to the global minimum is guaranteed for $L$-smooth convex functions, provided the learning rate is sufficiently small.