Offline Reinforcement Learning

Offline RL uses previously collected data without any additional data collection.

Offline RL is always Off-Policy, by definition. However, Off-Policy doesn’t necessarily mean offline RL. So offline RL is extra hard, because it can suffer from exploration more easily.

Techniques:

• DDPG
• CQL
• BCQ

The problem:

We have a Replay buffer that we are learning from: $\mathcal{D}=\{(s,a,r,s^{\prime }{)}_1,\dots \}$ We use this replay buffer to do our Bellman Expectation Backup $Q^{\pi }(s_t,a_t)\leftarrow r_t+Q^{\pi }(s_{t+1},a_{t+1})$ However,

https://chatgpt.com/share/687d9452-1790-8002-adc8-8efc3d707abd

But isn't this a problem in online RL as well???

Yes, but In online Q-learning, you can actually try those overestimated actions during exploration.

Say the Q-network hallucinates that $Q(s,a^{\prime })$ is high. The policy tries $a^{\prime }$ (via ε-greedy). If the reward is actually low, the Q-value is corrected: $Q(s,a^{\prime })\leftarrow r+\gamma {\max }_{a^{\prime \prime }}Q(s^{\prime },a^{\prime })$

🔁 So exploration and environment feedback act as a self-correcting mechanism.

So like it gets corrected. But in offline RL, you don’t have the benefit of controlling your exploration. You don’t get GPI.

That’s why conservative offline RL methods (like CQL, BCQ, etc.) try to stay close to the dataset, rather than trust the raw ${\max }_{a}Q(s,a)$ blindly.

So why do people try offline RL??

Real-world data is expensive or dangerous to collect. Logged data already exists. Think about the internet. So we want to figure out how to learn in an offline setting.