Generalised Advantage Estimation

reinforcement-learning

Definition

Generalised Advantage Estimation

Generalised advantage estimation (GAE) is an advantage estimator that smooths temporal-difference errors into an exponentially weighted, discounted sum. Given a policy with value estimate , define the temporal-difference error

the critic’s one-step surprise: how much better the transition with reward was than predicted. GAE with parameter is

Equivalently, in the recursive form used in implementations,

The value target is recovered as .

trades bias against variance

The two endpoints of pin down the trade:

• : the sum collapses to , the one-step temporal-difference error. Only one transition and a value lookup are involved, so variance is low; but the future beyond is replaced by the critic’s , whose error flows straight into the estimate, so it inherits the critic’s bias.
• : the sum telescopes to , the Monte Carlo advantage. The full realised return replaces every critic estimate beyond , so the estimate is unbiased; but aggregates many noisy rewards, so its variance grows with the horizon.

Intermediate keeps every TD error in the sum but shrinks the weight of distant ones via , leaning on the critic for the far future and on sampled rewards for the near future — the standard bias-variance tradeoff on the advantage estimator.

Derivation

The return satisfies the recursive identity , so telescoping

gives

where is the -step return. GAE replaces the hard cutoff at step with an exponential decay on the TD errors, which down-weights distant, noisier corrections while keeping every horizon in the mix.