Activation

machine-learning deep-learning analysis

Definition

Activation

In an artificial neural network, the activation function $\sigma (\cdot )$ is a non-linear mapping applied to the weighted sum of a neuron’s inputs (the pre-activation). Formally, for a pre-activation value $z={\mathbf{w}}^{\top }\mathbf{x}+b$, the output activation $a$ is:

$a=\sigma (z)$

Common

Sigmoid

Definition

Sigmoid Function

The sigmoid function (specifically the logistic function) is an S-shaped function that maps any real-valued input to the interval $(0,1)$. Formally:

$$\sigma (z)=\frac{1}{1+e^{-z}}=\frac{e^z}{1+e^z}$$

Link to original

ReLU

Definition

Rectified Linear Unit

The Rectified Linear Unit (ReLU) is a non-linear activation function that outputs the input directly if it is positive, and zero otherwise. Formally:

$f(z)=\max (0,z)=\{\begin{array}{ll}z& \text{if }z>0\\ 0& \text{otherwise}\end{array}$

Link to original