Policy Gradient Methods

Actor-Critic Methods

The point of actor critic methods is to decouple the gradient update from the q-function update.

• Policy-based methods directly optimize the policy but have high variance
• Value-based methods estimate values but don’t give a policy

Unlike Q-Learning (value-based methods), which directly attempts to learn the optimal Q-function, actor-critic methods aim to learn the Q-function corresponding to the current parameterized policy ${\pi }_{\theta }(a|s)$, which must obey the equation $Q^{\pi }(s,a)={\mathbb{E}}_{s^{\prime },r\sim p(\cdot |s,a)}[r+\gamma {\mathbb{E}}_{a^{\prime }\sim \pi (\cdot |s^{\prime })}{Q}^{\pi }(s^{\prime },a^{\prime })]$

Methods:

• A2C
• A3C
• DDPG
• Soft Actor-Critic