Actor-Critic Methods

Deep Deterministic Policy Gradient (DDPG)

More sample efficient. Though mostly replaced by TD3 since its just objectively better. DDPG can be thought of as being DQN for continuous action spaces.

Why is it called DDPG?

Because the policy that is learned is deterministic.

Resources

• https://spinningup.openai.com/en/latest/algorithms/ddpg.html
  • Implementation here
• Lecture 5: DDPG and SAC from Deep RL Foundations, slides here

Algorithms like DDPG and Q-Learning are off-policy, so they are able to reuse old data very efficiently. They gain this benefit by exploiting Bellman’s equations for optimality, which a Q-function can be trained to satisfy using any environment interaction data.

DDPG interleaves learning an approximator for $Q^{\ast }(s,a)$ with learning an approximator to $a(s)$.

Mean Squared Bellman Error: $L(\varphi ,\mathcal{D})=\underset{(s,a,r,s^{\prime },d)\sim \mathcal{D}}{E}\left[(Q_{\varphi }(s,a)-\left(r+\gamma (1-d){\max }_{a^{\prime }}{Q}_{{\varphi }_{target}}(s^{\prime },a^{\prime })\right){)}^2\right]$

• Notice that we take the max, so it looks like a Bellman Optimality Backup

In practice however, we deal with continuous action spaces, taking the max is not so obvious. So we learn a policy network ${\mu }_{\theta }$ to predict how to maximize Q: $L(\varphi ,\mathcal{D})=\underset{(s,a,r,s^{\prime },d)\sim \mathcal{D}}{E}\left[(Q_{\varphi }(s,a)-\left(r+\gamma (1-d)Q_{{\varphi }_{\text{targ}}}(s^{\prime },{\mu }_{{\theta }_{\text{targ}}}(s^{\prime }))\right){)}^2\right]$

For updating the policy ${\mu }_{\theta }$, we just do gradient ascent to maximize $Q$: ${\max }_{\theta }\underset{s\sim \mathcal{D}}{E}\left[Q_{\varphi }(s,{\mu }_{\theta }(s))\right]$

Notes on Implementation

You will notice that the network will learn an MLP to predict both mean and variance.

Related

• Not to be confused with DDPM