Reinforcement Learning

Entropy-Regularized Reinforcement Learning

In standard reinforcement learning, the agent maximizes the expected return:

$$\underset{\pi }{\max }{\mathbb{E}}_{\pi }\left[\sum _{t}r(s_t,a_t)\right]$$

In entropy regularized reinforcement learning, the objective becomes:

$$\underset{\pi }{\max }{\mathbb{E}}_{\pi }\left[\sum _t\left(r(s_t,a_t)+\alpha \mathcal{H}(\pi (\cdot |s_t))\right)\right]$$

where:

• $\mathcal{H}(\pi (\cdot |s_t))=-{\sum }_a\pi (a|s_t)\log \pi (a|s_t)$ is the entropy of the policy at state $s_t$
• $\alpha$ is a temperature coefficient that controls the trade-off between reward and entropy

Controlling exploration

You can think of this as the way to get exploration when we are working with continuous action space.

Related

• n-step Reinforcement Learning