Policy Gradient Methods

Trust Region Policy Optimization (TRPO)

TRPO updates policies by taking the largest step possible to improve performance, while satisfying a special constraint on how close the new and old policies are allowed to be.

Resources

• https://spinningup.openai.com/en/latest/algorithms/trpo.html
  • Implementation here
• OG paper https://proceedings.mlr.press/v37/schulman15.html

In Vanilla Policy Gradient, we just do a gradient update of the weights. ${\pi }_{\theta }(s|a)$ can change drastically between each step if you think about the parameter landscape.

TRPO nicely avoids this kind of collapse, and tends to quickly and monotonically improve performance by adding a KL Divergence constraint, which forces ${\pi }_{new}(a_t|s_t)$ to be similar to ${\pi }_{old}(a_t|s_t)$.

I was thinking looking at the VPG spinning up implementation that they would just use log_p - old_log_p, and then can use this alpha as a regularization term. However, it doesn’t seem like they do that.

${\theta }_{k+1}=\arg {\max }_{\theta }\mathcal{L}({\theta }_k,\theta )\text{s.t.}{\bar{D}}_{KL}(\theta ||{\theta }_k)\le \delta$ The goal is the find the best parameters that maximizes the surrogate advantage that satisfies the KL divergence constraint.

The surrogate advantage: $\mathcal{L}({\theta }_k,\theta )={\mathbb{E}}_{s,a\sim {\pi }_{{\theta }_k}}\left[\frac{{\pi }_{\theta }(a|s)}{{\pi }_{{\theta }_k}(a|s)}{A}^{{\pi }_{{\theta }_k}}(s,a)\right]$

And KL Divergence constraint: ${\bar{D}}_{KL}(\theta ||{\theta }_k)={\mathbb{E}}_{s\sim {\pi }_{{\theta }_k}}[D_{KL}\left({\pi }_{\theta }(\cdot |s)||{\pi }_{{\theta }_k}(\cdot |s)\right)]$

Related

• PPO