Convex Function

statistics analysis optimisation

Definition

Convex Function

A function $f:X\to \mathbb{R}$ defined on a convex set $X$ is convex if the line segment between any two points on the graph of the function lies above or on the graph. Formally, for all $x_1,x_2\in X$ and any $\lambda \in [0,1]$:

$f(\lambda x_1+(1-\lambda )x_2)\le \lambda f(x_1)+(1-\lambda )f(x_2)$

Significance in Optimisation

Global Optimality: For convex functions, any local minimum is also a global minimum. This property is fundamental for the convergence of gradient descent and the efficiency of training models like SVMs and Logistic Regression.

Jensen’s Inequality: A key result in probability and analysis, stating that for a convex function $f$ and random variable $X$, $f(\mathbb{E}[X])\le \mathbb{E}[f(X)]$. This is frequently used in the derivation of learning bounds and variational inference.