Policy Gradient

reinforcement-learning

Definition

Policy Gradient

Given a parameterised stochastic policy ${\pi }_{\theta }(a\mid s)$, the policy gradient is the gradient of the expected return

$$J(\theta )={\mathbb{E}}_{\tau \sim {\pi }_{\theta }}[R(\tau )]$$

with respect to the policy parameters:

$${\nabla }_{\theta }J(\theta ).$$

By the policy gradient theorem, it can be estimated as

$${\nabla }_{\theta }J(\theta )={\mathbb{E}}_t\left[{\nabla }_{\theta }\log {\pi }_{\theta }(a_t\mid s_t)A_t\right],$$

where $A_t$ is an advantage estimate. Therefore, actions with positive advantage are made more likely, while actions with negative advantage are made less likely.