Off-Policy Methods

Importance Sampling

Almost all off-policy methods utilize Importance Sampling.

Importance Sampling is a general technique for estimating expected values under one distribution given samples from another. We can use the following approximation

${\mathbb{E}}_{x-q}[f(x)]={\int }_{x}q(x)f(x)dx$ $\approx \sum _{i=1}^n\frac{q(x_i)}{p(x_i)}f(x_i)$

From ChatGPT

In off-policy reinforcement learning (RL), we use data generated by a behavior policy (say, $\beta$) to improve a target policy ($\pi$).

• The issue is that the distribution of data (state-action pairs) comes from $\beta$, but we’re evaluating or improving $\pi$
• This distribution mismatch needs to be corrected

This is what we do with importance sampling! For each sample $(s,a)$, we weight its contribution by the importance ratio: $w(s,a)=\frac{\pi (a\mid s)}{\beta (a\mid s)}$

Suppose:

• β always takes action A.
• π takes action A only half the time and action B the other half.

Without importance sampling, we would overestimate the frequency of action A in π.

F1TENTH

Learning this under the Particle Filter lecture. Basically, if the distribution you want to sample from is normal, you know what values to use. However, how would you do it for any distribution?

We can use the importance sampling principle.