Hyperbolic Tangent

math analysis deep-learning

Definition

Hyperbolic Tangent

The hyperbolic tangent ($\tanh$) is a non-linear activation function that maps real-valued inputs to the interval $(-1,1)$. Formally:

$\tanh (z)=\frac{e^z-e^{-z}}{e^z+e^{-z}}=2\sigma (2z)-1$

where $\sigma (\cdot )$ is the sigmoid function.

Properties

Zero-Centring: Unlike the sigmoid function, $\tanh$ is zero-centred, meaning its output range is symmetric around the origin. This property often leads to more efficient gradient-based optimisation during the training of deep artificial neural networks.

Vanishing Gradients: Similar to the sigmoid, $\tanh$ saturates for large absolute values of $z$, where the derivative ${\tanh }^{\prime }(z)=1-{\tanh }^2(z)$ approaches zero. This can impede training in very deep architectures, leading to the preference for ReLU.