High-Dimensional Continuous Control Using Generalized Advantage Estimation

Heard from the spinning up

• “For a more detailed treatment of this topic, you should read the paper on Generalized Advantage Estimation (GAE), which goes into depth about different choices of ${\Phi }_t$ in the background sections.”
• https://spinningup.openai.com/en/latest/spinningup/rl_intro3.html#id16

There are a few ${\varphi }_t$ that we could choose: ${\varphi }_t=R(\tau )$ ${\varphi }_t=R(s_t,a_t,s_{t+1})$ ${\varphi }_t=V(s)-{\hat{R}}_t$

• where $\hat{R}$ is a Reward-to-go ${\varphi }_t=Q(s,a)-V(s)$
• This is know as the Advantage Function

https://arxiv.org/pdf/1506.02438

Citation

(Schulman et al., 2015)

Schulman, J., Moritz, P., Levine, S., Jordan, M., & Abbeel, P. (2015). High-Dimensional Continuous Control Using Generalized Advantage Estimation. https://arxiv.org/pdf/1506.02438