Support Vector Machine

machine-learning classification regression optimisation

Definition

Support Vector Machine

A Support Vector Machine (SVM) is a supervised learning model that identifies the optimal hyperplane $w^{\top }x+b=0$ that separates classes with the maximum possible margin. Formally, given a dataset $D=\{(x_i,y_i){\}}_{i=1}^n$ with $y_i\in \{-1,+1\}$, the SVM solves the constrained optimisation problem:

${\min }_{\mathbf{w},b}\frac{1}{2}\Vert \mathbf{w}{\Vert }^2\text{subject to}{y}_i({\mathbf{w}}^{\top }{\mathbf{x}}_i+b)\ge 1,\forall i$

Structural Framework

Support Vectors: These are the critical data points that lie exactly on the margin boundaries. The optimal hyperplane is determined exclusively by these points; removing any other data points from the set would not change the resulting model.

Hard-margin SVM: The original formulation which assumes the data is strictly linearly separable. It enforces the constraint that no data points may reside within the margin or be misclassified.

Soft-margin SVM: An extension for non-separable or noisy data that introduces slack variables ${\xi }_i$ and a regularisation parameter $C$, utilising the hinge loss to penalise violations of the margin.

Kernel SVM: By applying the kernel trick, SVMs can identify maximum-margin hyperplanes in high-dimensional feature spaces, enabling the classification of non-linearly separable data in the original input space.