Imitation Learning

Behavior Cloning (BC)

Imitation Learning is often formulated as behavior cloning, which uses supervised learning to learn a policy ${\pi }_{\theta }$ parameterized by $\theta$ to maximize the log-likelihood of actions in a dataset $D$:

MLE formulation: ${\max }_{\theta }{\mathbb{E}}_{(s,a)\sim D}[\log {\pi }_{\theta }(a|s)]$

• As seen in What Matters for Batch Online Reinforcement Learning in Robotics papers
• See Log Likehood if confused about where log is coming from
• More intelligent BC can be done with
• Maximizing $\pi (a|s)$ just means: “make the model put high probability on the demonstrated (expert) actions.”
  • In discrete spaces → cross-entropy.
    • Ground truth action is One-Hot Encoded, i.e. $a=[0,0,1,0]$
  • In continuous spaces → likelihood under a distribution (e.g., Gaussian).
    • Ground truth action is continuous, i.e. $a=[0.2,0.6,0.4]$ given some state, for each dimension we learn a gaussian

Regression formulation: ${\min }_{\theta }{\mathbb{E}}_{(s,a)\sim D}||{\pi }_{\theta }(s)-a|{|}^2$

• As seen in Robomimic paper
• With this formulation, we lose the ability to learn uncertainty

In pi0, instead cross entropy, they use Flow Matching: ${\min }_{\theta }||v_{\theta }(s,a)-u(s,a)|{|}^2$

• It looks like regression, but the model predicts a velocity field

If the policy “just always outputs one action,” it’s probably overfitting or collapsing due to data bias or model simplicity.

KL divergence shows up when comparing your learned policy to another (e.g., expert or prior) distribution.

What happens if we drop the log?

Like why can’t we maximize $\mathbb{E}[{\pi }_{\theta }(a|s)]$ ? If you maximize the raw probability, it can lead to weird behaviors:

• For distributions like Gaussians, the “probability” can be greater than 1, and maximizing it directly isn’t a proper scoring rule
• It loses the property of penalizing low-probability assignments strongly.
• You don’t recover MLE or Cross-Entropy Loss

Assume that our policy learned is just a univariate gaussian distribution, so we only need to learn $\mu ,\sigma$. If you don’t have the log, it’s going to collapse to $\sigma =0$.

• If you add the log, it’s the Log Likelihood

Related

• Implicit Behavioral Cloning