Gradient Checking

machine-learning optimisation calculus

Definition

Gradient Checking

Gradient checking is a debugging technique used to verify the correctness of an analytical gradient implementation (typically computed via backpropagation). It compares the analytical gradient $\nabla J(\theta )$ with a numerical approximation derived from the limit definition of the derivative.

Numerical Approximation

The approximation is calculated using the two-sided finite difference formula for a small $ϵ$ (e.g., $10^{-7}$):

$\frac{\partial J}{\partial {\theta }_i}\approx \frac{J({\theta }_1,\dots ,{\theta }_i+ϵ,\dots ,{\theta }_n)-J({\theta }_1,\dots ,{\theta }_i-ϵ,\dots ,{\theta }_n)}{2ϵ}$

Verification Criterion: The implementation is considered correct if the relative difference between the analytical gradient $G_a$ and the numerical gradient $G_n$ is sufficiently small:
$\frac{\Vert G_a-G_n\Vert }{\Vert G_a\Vert +\Vert G_n\Vert }<\text{threshold}$
This process ensures that the complex chain of partial derivatives in deep neural architectures is mathematically sound before proceeding with large-scale training.