Basis Function

mathematics machine-learning regression

Definition

Basis Function

A basis function is an element of a set of functions used to represent or approximate more complex mappings via linear combinations. Formally, a function $f(x)$ is represented as:

$f(x)={\sum }_{j=0}^M{w}_j{\varphi }_j(x)={\mathbf{w}}^{\top }\Phi (x)$

where $\{{\varphi }_0,{\varphi }_1,\dots ,{\varphi }_M\}$ is the set of basis functions and $\mathbf{w}$ is the vector of coefficients.

Applications in Regression

Basis functions are utilised to extend linear models to capture non-linear relationships while maintaining a linear optimisation framework.

Polynomial Basis: In polynomial regression, the basis functions are powers of the input: ${\varphi }_j(x)=x^j$. This expands the feature space to allow for curvilinear fits.

Radial Basis Functions: In models like the RBF network, basis functions depend on the distance from a centre ${\mathbf{c}}_j$: ${\varphi }_j(\mathbf{x})=\exp (-\gamma \Vert \mathbf{x}-{\mathbf{c}}_j{\Vert }^2)$.

Kernel Methods: The kernel trick can be viewed as an implicit expansion into an infinite-dimensional feature space defined by a set of basis functions, where the kernel $k(x,x^{\prime })$ represents the inner product of the basis expansions.