Advantage Function

reinforcement-learning

Definition

Advantage Function

Let $\pi$ be a policy, $V^{\pi }(s)={\mathbb{E}}_{\pi }[G_t\mid s_t=s]$ its on-policy value function, and $Q^{\pi }(s,a)={\mathbb{E}}_{\pi }[G_t\mid s_t=s,a_t=a]$ its on-policy action-value function.

The advantage function of action $a$ in state $s$ is

$$A^{\pi }(s,a)=Q^{\pi }(s,a)-V^{\pi }(s).$$

It measures how much better it is to take action $a$ in $s$ than to follow $\pi$ on average from $s$. Positive advantage means $a$ beats the policy’s typical return; negative advantage means it underperforms.

Because $V^{\pi }(s)={\mathbb{E}}_{a\sim \pi (\cdot \mid s)}[Q^{\pi }(s,a)]$, the advantage is centred at zero:

$${\mathbb{E}}_{a\sim \pi (\cdot \mid s)}[A^{\pi }(s,a)]=0.$$

Subtracting $V^{\pi }(s)$ as a baseline removes the part of the return predictable from the state alone, which reduces the variance of policy-gradient estimators without introducing bias.

Estimators

Monte Carlo Advantage

Definition

Monte Carlo Advantage Estimation

Monte Carlo advantage estimation estimates the advantage of the action actually taken by subtracting a value baseline from a single sampled return:

$$A_t^{\mathrm{MC}}=G_t-V^{\pi }(s_t),$$

where $G_t$ is the realised discounted return from time $t$. It stands in for the true advantage $A^{\pi }(s_t,a_t)=Q^{\pi }(s_t,a_t)-V^{\pi }(s_t)$ by using the sample $G_t$ in place of the intractable expectation $Q^{\pi }(s_t,a_t)={\mathbb{E}}_{\pi }[G_t\mid s_t,a_t]$.

Link to original

Generalised Advantage

Definition

Generalised Advantage Estimation

Generalised advantage estimation (GAE) is an advantage estimator that smooths temporal-difference errors into an exponentially weighted, discounted sum. Given a policy $\pi$ with value estimate $V^{\pi }$, define the temporal-difference error

$${\delta }_t=r_t+\gamma V^{\pi }(s_{t+1})-V^{\pi }(s_t),$$

the critic’s one-step surprise: how much better the transition $s_t\to s_{t+1}$ with reward $r_t$ was than $V^{\pi }(s_t)$ predicted. GAE with parameter $\lambda \in [0,1]$ is

$$A_t^{\mathrm{GAE}(\gamma ,\lambda )}=\sum _{l=0}^{\infty }(\gamma \lambda {)}^l{\delta }_{t+l}.$$

Equivalently, in the recursive form used in implementations,

$$A_t^{\mathrm{GAE}(\gamma ,\lambda )}={\delta }_t+\gamma \lambda A_{t+1}^{\mathrm{GAE}(\gamma ,\lambda )}.$$

The value target is recovered as $R_t=A_t^{\mathrm{GAE}}+V^{\pi }(s_t)$.

Link to original