Offline Reinforcement Learning

Offline RL uses previously collected data without any additional data collection.

Offline RL is essentially a tug-of-war between behavioral regularization and value maximization.

Resources / readings:

• Offline RL or BC? https://bair.berkeley.edu/blog/2022/04/25/rl-or-bc/

Techniques:

• IQL
• CQL
• Cal-QL
• BCQ
• Flow Q-Learning

Notation below from Conservative Q-Learning. We have a Replay buffer that we are learning from: $\mathcal{D}=\{(s,a,r,s^{\prime }{)}_1,\dots \}$ Policy Evaluation step via Bellman Expectation Backup ${\hat{Q}}_{k+1}\leftarrow \arg {\min }_Q{\mathbb{E}}_{s,a,s^{\prime }\sim \mathcal{D}}\left[{\left(r(s,a)+\gamma {\mathbb{E}}_{a^{\prime }\sim {\hat{\pi }}_k(\cdot |s^{\prime })}\left[{\hat{Q}}_k(s^{\prime },a^{\prime })\right]-Q(s,a)\right)}^2\right]$

• Find the $Q$ that minimizes this Bellman Error

Policy improvement

$${\hat{\pi }}_{k+1}\leftarrow \arg \underset{\pi }{\max }{\mathbb{E}}_{s\sim \mathcal{D},a\sim {\pi }_k(\cdot |s)}\left[{\hat{Q}}_{k+1}(s,a)\right]$$

Notice that we train our Q from a behavior policy out of $\mathcal{D}$ by sampling these $(s,a,r,s^{\prime })$ from D. and then follow a behavior policy ${\pi }_{\beta }$ that tries to maximize $Q$. However, this can get biased towards actions on Q that are OOD.

As a result, offline RL tends to overestimate the Q values.

• https://chatgpt.com/share/687d9452-1790-8002-adc8-8efc3d707abd

But isn't this a problem in online RL as well???

Yes, but In online Q-learning, you can actually try those overestimated actions during exploration.

Say the Q-network hallucinates that $Q(s,a^{\prime })$ is high. The policy tries $a^{\prime }$ (via ε-greedy). If the reward is actually low, the Q-value is corrected: $Q(s,a^{\prime })\leftarrow r+\gamma {\max }_{a^{\prime \prime }}Q(s^{\prime },a^{\prime })$

🔁 So exploration and environment feedback act as a self-correcting mechanism.

So like it gets corrected. But in offline RL, you don’t have the benefit of controlling your exploration. You don’t get GPI.

That’s why conservative offline RL methods (like CQL, BCQ etc.) try to stay close to the dataset, rather than trust the raw ${\max }_{a}Q(s,a)$ blindly.

So why do people try offline RL??

Real-world data is expensive or dangerous to collect. Logged data already exists. Think about the internet. So we want to figure out how to learn in an offline setting.

Offline RL vs Off-Policy RL?

Offline RL is always Off-Policy, by definition. However, Off-Policy doesn’t necessarily mean offline RL. Offline RL is extra hard, because it cannot explore, fixed offline dataset (unless you do something like Batch Online RL).