Advantage Estimate

reinforcement-learning

Definition

Advantage Estimate

An advantage estimate ${\hat{A}}_t$ is an empirical approximation of the advantage function $A^{\pi }(s_t,a_t)$, computed from sampled trajectories.

Common estimators:

• TD residual: ${\hat{A}}_t={\delta }_t=r_t+\gamma V(s_{t+1})-V(s_t)$.
• GAE: ${\hat{A}}_t={\sum }_{l=0}^{\infty }(\gamma \lambda {)}^l{\delta }_{t+l}$, interpolating between low-variance ($\lambda =0$, TD) and low-bias ($\lambda =1$).

The estimate enters the policy gradient as the weight multiplying ${\nabla }_{\theta }\log {\pi }_{\theta }(a_t\mid s_t)$.