Multi-Layer Perceptron

machine-learning deep-learning

Definition

Multi-Layer Perceptron

A Multi-Layer Perceptron (MLP) is a feedforward artificial neural network consisting of an input layer, at least one hidden layer, and an output layer. Formally, the activation vector ${\mathbf{a}}^{(l)}$ of layer $l$ is defined by the recursive transformation:

${\mathbf{a}}^{(l)}=\sigma ({\mathbf{s}}^{(l)})=\sigma (W^{(l-1)}{\mathbf{a}}^{(l-1)}+{\mathbf{b}}^{(l-1)})$

where:

• ${\mathbf{s}}^{(l)}$ is the pre-activation vector of layer $l$.
• $W^{(l-1)}$ is the weight matrix connecting layer $l-1$ to layer $l$.
• ${\mathbf{b}}^{(l-1)}$ is the bias vector of the transformation.
• $\sigma (\cdot )$ is a non-linear activation function.

Functional Expressivity

Universal Approximation: According to the Universal Function Approximation Theorem, an MLP with a single hidden layer containing a finite number of neurons can approximate any continuous function on compact subsets of ${\mathbb{R}}^n$, provided the activation function is non-linear and bounded.

Linearity Collapse: Without the inclusion of non-linear activation functions, any stack of multiple layers collapses into a single linear transformation, as the composition of affine mappings is itself affine: $W_2(W_1\mathbf{x}+{\mathbf{b}}_1)+{\mathbf{b}}_2=W^{\prime }\mathbf{x}+{\mathbf{b}}^{\prime }$.