Rectified Linear Unit

machine-learning deep-learning analysis

Definition

Rectified Linear Unit

The Rectified Linear Unit (ReLU) is a non-linear activation function that outputs the input directly if it is positive, and zero otherwise. Formally:

$f(z)=\max (0,z)=\{\begin{array}{ll}z& \text{if }z>0\\ 0& \text{otherwise}\end{array}$

Operational Advantages

Gradient Flow: Unlike saturating activation functions like Sigmoid, ReLU does not suffer from the vanishing gradient problem for positive inputs, as the gradient is constant ($\frac{\partial f}{\partial z}=1$). This property is fundamental for the successful training of deep artificial neural networks.

Sparsity: Since any input $z\le 0$ results in a zero activation, ReLU naturally induces sparsity in the hidden layers, which can lead to more robust and computationally efficient representations.