Policy Gradient Theorem

reinforcement-learning

Definition

Policy Gradient Theorem

Let ${\pi }_{\theta }$ be a parameterised stochastic policy and let $J(\theta )={\mathbb{E}}_{\tau \sim {\pi }_{\theta }}[G_0]$ be the . The expected return gradient of $J$ with respect to $\theta$ is

$${\nabla }_{\theta }J(\theta )={\mathbb{E}}_{\tau \sim {\pi }_{\theta }}\left[\sum _{t=0}^{\infty }{\nabla }_{\theta }\log {\pi }_{\theta }(a_t\mid s_t){\Psi }_t\right]$$

where ${\Psi }_t$ is any baseline that depends only on $s_t$. Common choices include the return $G_t$, the action-value $Q^{\pi }(s_t,a_t)$, or the advantage $A^{\pi }(s_t,a_t)$.

The theorem reduces the problem of differentiating through environment dynamics to differentiating only through the policy’s own log-probabilities, enabling gradient-based policy improvement without a model of $P(s^{\prime }\mid s,a)$.