Advantage-Weighted Regression (AWR)

Saw this from the Batch Online RL paper.

${\theta }^{\ast }=\arg {\max }_{\theta }{\mathbb{E}}_{(s,a)\sim D}\left[e^{\beta (Q(s,a)-V(s))}\log {\pi }_{\theta }(a|s)\right]$

• So instead of naive BC, we reweigh the dataset based on this advantage

AWR learns a policy pi(a∣s) by supervised learning on a dataset of (s,a) pairs, but weights each action by its advantage:

Learn a policy that imitates actions with high advantage, and suppresses actions with low advantage.

Related

• Policy Extraction