DeepMimic: Example-Guided Deep Reinforcement Learning of Physics-Based Character Skills

Core idea: “Minimize tracking error”

Very FUNDAMENTAL paper, really set the path for making locomation work.

• Limitation is that it’s a single-motion policy, later papers (and currently) we try to address this

Mentioned from BeyondMimic.

DeepMimic project page:

• https://xbpeng.github.io/projects/DeepMimic/index.html

Blog https://bair.berkeley.edu/blog/2018/04/10/virtual-stuntman/

Where did they get the mocap from?

“Each skill is learned from approximately 0.5-5s of mocap data collected from http://mocap.cs.cmu.edu and http://mocap.cs.sfu.ca.”

The rewards:

$$R_{It}=w_p{R}_{pt}+w_v{R}_{vt}+w_e{R}_{et}+w_c{R}_{ct}$$

• And the gains are slightly tuned

Is there normalization applied to those rewards?

Where each reward is something along the lines of

$$r_t^p=\exp \left[-2\Vert {\hat{q}}_t-q_t{\Vert }^2\right].$$

Some insights (mostly came from blog):

• Reference state initialization is very important to get it to work
• Early termination
  • ” This is analogous to the class imbalance problem encountered by other methodologies such as supervised learning.”