Gaussian Radial Basis Function

machine-learning statistics deep-learning

Definition

Gaussian Radial Basis Function

The Gaussian Radial Basis Function (RBF) kernel is a popular stationary kernel that measures the similarity between points based on their Euclidean distance. Formally, for a free parameter $\sigma >0$:

$k(\mathbf{x},{\mathbf{x}}^{\prime })=\exp \left(-\frac{\Vert \mathbf{x}-{\mathbf{x}}^{\prime }{\Vert }^2}{2{\sigma }^2}\right)$

Alternatively, it is often parameterised using $\gamma =\frac{1}{2{\sigma }^2}$, yielding $k(\mathbf{x},{\mathbf{x}}^{\prime })=\exp (-\gamma \Vert \mathbf{x}-{\mathbf{x}}^{\prime }{\Vert }^2)$.

Infinite Dimensionality

The RBF kernel corresponds to a feature map $\varphi$ into an infinite-dimensional Hilbert space. It can be viewed as an infinite sum of polynomial kernels of all possible degrees, allowing the SVM to approximate arbitrary non-linear decision boundaries.

Generalisation

Locality: The kernel value decreases exponentially with the squared distance, meaning that only nearby points significantly influence the local prediction. This makes the model highly flexible but sensitive to the selection of the bandwidth parameter $\sigma$; excessively small values lead to overfitting (memorising individual points), while excessively large values result in underfitting.