Generalised Advantage Estimation

reinforcement-learning

Definition

Generalised Advantage Estimation

Generalised advantage estimation (GAE) is an advantage estimator that smooths temporal-difference errors into an exponentially weighted, discounted sum. Given a policy $\pi$ with value estimate $V^{\pi }$, define the temporal-difference error

$${\delta }_t=r_t+\gamma V^{\pi }(s_{t+1})-V^{\pi }(s_t),$$

the critic’s one-step surprise: how much better the transition $s_t\to s_{t+1}$ with reward $r_t$ was than $V^{\pi }(s_t)$ predicted. GAE with parameter $\lambda \in [0,1]$ is

$$A_t^{\mathrm{GAE}(\gamma ,\lambda )}=\sum _{l=0}^{\infty }(\gamma \lambda {)}^l{\delta }_{t+l}.$$

Equivalently, in the recursive form used in implementations,

$$A_t^{\mathrm{GAE}(\gamma ,\lambda )}={\delta }_t+\gamma \lambda A_{t+1}^{\mathrm{GAE}(\gamma ,\lambda )}.$$

The value target is recovered as $R_t=A_t^{\mathrm{GAE}}+V^{\pi }(s_t)$.

$\lambda$ trades bias against variance

The two endpoints of $\lambda \in [0,1]$ pin down the trade:

• $\lambda =0$: the sum collapses to $A_t^{\mathrm{GAE}}={\delta }_t$, the one-step temporal-difference error. Only one transition and a value lookup are involved, so variance is low; but the future beyond $s_{t+1}$ is replaced by the critic’s $V^{\pi }(s_{t+1})$, whose error flows straight into the estimate, so it inherits the critic’s bias.
• $\lambda =1$: the sum telescopes to $A_t^{\mathrm{GAE}}=G_t-V^{\pi }(s_t)$, the Monte Carlo advantage. The full realised return replaces every critic estimate beyond $V^{\pi }(s_t)$, so the estimate is unbiased; but $G_t$ aggregates many noisy rewards, so its variance grows with the horizon.

Intermediate $\lambda$ keeps every TD error in the sum but shrinks the weight of distant ones via $(\gamma \lambda {)}^l$, leaning on the critic for the far future and on sampled rewards for the near future — the standard bias-variance tradeoff on the advantage estimator.

Derivation

The return satisfies the recursive identity $G_t=r_t+\gamma G_{t+1}$, so telescoping

$${\delta }_t=r_t+\gamma V^{\pi }(s_{t+1})-V^{\pi }(s_t)$$

gives

$$\sum _{l=0}^{k-1}{\gamma }^l{\delta }_{t+l}=G_{t:t+k}-V^{\pi }(s_t),$$

where $G_{t:t+k}={\sum }_{l=0}^{k-1}{\gamma }^l{r}_{t+l}+{\gamma }^k{V}^{\pi }(s_{t+k})$ is the $k$-step return. GAE replaces the hard cutoff at step $k$ with an exponential decay ${\lambda }^l$ on the TD errors, which down-weights distant, noisier corrections while keeping every horizon in the mix.