Advantage Function

reinforcement-learning

Definition

Advantage Function

Let be a policy, its on-policy value function, and its on-policy action-value function.

The advantage function of action in state is

It measures how much better it is to take action in than to follow on average from . Positive advantage means beats the policy’s typical return; negative advantage means it underperforms.

Because , the advantage is centred at zero:

Subtracting as a baseline removes the part of the return predictable from the state alone, which reduces the variance of policy-gradient estimators without introducing bias.

Estimators

Monte Carlo Advantage

Definition

Monte Carlo Advantage Estimation

Monte Carlo advantage estimation estimates the advantage of the action actually taken by subtracting a value baseline from a single sampled return:

where is the realised discounted return from time . It stands in for the true advantage by using the sample in place of the intractable expectation .

Link to original

Generalised Advantage

Definition

Generalised Advantage Estimation

Generalised advantage estimation (GAE) is an advantage estimator that smooths temporal-difference errors into an exponentially weighted, discounted sum. Given a policy with value estimate , define the temporal-difference error

the critic’s one-step surprise: how much better the transition with reward was than predicted. GAE with parameter is

Equivalently, in the recursive form used in implementations,

The value target is recovered as .

Link to original