Advantage Function

reinforcement-learning

Definition

Advantage Function

Let $\pi$ be a policy, let $V^{\pi }(s)$ be the on-policy value function, and let $Q^{\pi }(s,a)$ be the on-policy action-value function.

The advantage function $A^{\pi }(s,a)$ quantifies how much better taking action $a$ in state $s$ is compared to following $\pi$ on average:

$$A^{\pi }(s,a)=Q^{\pi }(s,a)-V^{\pi }(s).$$

Using the advantage instead of the raw return as a learning signal reduces variance without introducing bias, since ${\mathbb{E}}_{a\sim \pi (\cdot \mid s)}[A^{\pi }(s,a)]=0$.