Temporal-Difference Error

reinforcement-learning

Definition

Temporal-Difference Error

The temporal-difference (TD) error is the difference between two successive estimates of the same value, one step apart. Given a policy $\pi$ with value estimate $V^{\pi }$, it is

$${\delta }_t=r_t+\gamma V^{\pi }(s_{t+1})-V^{\pi }(s_t).$$

The term $r_t+\gamma V^{\pi }(s_{t+1})$ is the one-step target: the reward just received, plus the discounted value of the state reached. Subtracting $V^{\pi }(s_t)$ asks how much that target disagrees with the critic’s current prediction. A positive ${\delta }_t$ means the transition was better than expected; a negative one means it was worse.

Relation to the Bellman equation

The TD error vanishes in expectation when $V^{\pi }$ is the true value function. Taking expectations and using $V^{\pi }(s)={\mathbb{E}}_{\pi }[r+\gamma V^{\pi }(s^{\prime })]$,

$${\mathbb{E}}_{\pi }[{\delta }_t\mid s_t=s]=0,$$

which is exactly the Bellman consistency condition for $V^{\pi }$. So ${\delta }_t$ is the residual of that equation on a single sampled transition: it is zero on average when the value is correct, and nonzero wherever the critic is wrong.

As a learning signal

The TD error is the basic update signal of temporal-difference learning. The value estimate is moved toward the target by a step proportional to ${\delta }_t$, which drives $V^{\pi }$ toward Bellman consistency without waiting for a full episode’s return. It also underlies advantage estimation: generalised advantage estimation builds its estimator from a discounted sum of TD errors.