Backpropagation

machine-learning optimisation calculus

Definition

Backpropagation

Backpropagation is the standard algorithm used to calculate the gradient of the loss function with respect to the weights of an artificial neural network. It is a specific application of the chain rule from calculus, enabling the efficient training of multi-layer architectures.

The process is divided into two distinct phases:

1. Forward Pass: The input is propagated through the network to compute the predicted output and the resulting loss.
2. Backward Pass: The error signal is propagated from the output layer back toward the input layer, calculating partial derivatives for each weight.

Algorithmic Procedure

Chain Rule Application: For a weight $w_{ij}^{(l)}$ connecting neuron $j$ in layer $l-1$ to neuron $i$ in layer $l$, the gradient is computed as:

$$\frac{\partial L}{\partial w_{ij}^{(l)}}=\frac{\partial L}{\partial a_i^{(l)}}\cdot \frac{\partial a_i^{(l)}}{\partial z_i^{(l)}}\cdot \frac{\partial z_i^{(l)}}{\partial w_{ij}^{(l)}}$$

where $z$ is the pre-activation and $a$ is the activation.

Weight Update: Once the gradients are computed, the parameters $\theta$ are updated using an optimisation algorithm such as gradient descent:
$\theta \leftarrow \theta -\eta {\nabla }_{\theta }L$

Computational Graph Example

Consider the function $f(x,y,z)=(x+y)z$. This can be decomposed into intermediate steps to demonstrate the local gradient calculations.

Forward Pass: Let $q=x+y$, then $f=qz$. Given $x=-2,y=5,z=-4$:

• $q=3$
• $f=-12$

Backward Pass: Calculating the derivatives via the chain rule:

• $\frac{\partial f}{\partial z}=q=3$
• $\frac{\partial f}{\partial q}=z=-4$
• $\frac{\partial f}{\partial x}=\frac{\partial f}{\partial q}\frac{\partial q}{\partial x}=-4\cdot 1=-4$
• $\frac{\partial f}{\partial y}=\frac{\partial f}{\partial q}\frac{\partial q}{\partial y}=-4\cdot 1=-4$

This demonstrates how the gradient with respect to the inputs is efficiently calculated by propagating the local gradients backward through the graph.