Actor-Critic Methods

Soft Actor-Critic (SAC)

Resources

• https://spinningup.openai.com/en/latest/algorithms/sac.html
  • Implementation here
• Original paper https://arxiv.org/pdf/1801.01290

We optimize the policy according to ${\max }_{\theta }{\mathbb{E}}_{s\sim \mathcal{D},\xi \sim \mathcal{N}}\left[{\min }_{j=1,2}{Q}_{{\varphi }_j}(s,{\tilde{a}}_{\theta }(s,\xi ))-\alpha \log {\pi }_{\theta }({\tilde{a}}_{\theta }(s,\xi )|s)\right]$

• $\tilde{a}$ is to represent that the action is sampled fresh from the policy, as opposed to taking the action $a$ from the replay buffer.

where ${\tilde{a}}_{\theta }(s,\xi )$ is a sample from the policy with Reparametrization Trick ${\tilde{a}}_{\theta }(s,\xi )=\tanh \left({\mu }_{\theta }(s)+{\sigma }_{\theta }(s)\odot \xi \right),\xi \sim \mathcal{N}(0,I)$

What / why Reparametrization Trick?

This was initially confusing to me, since in VPG, we also sample the action, yet we don’t seem to need the reparametrization trick.

The issue here is that we want to figure out how change our parameters $\theta$ to maximize $Q_{\varphi }$. However, the gradients just stop propagating once we pass $a\sim {\pi }_{\theta }(a\mid s)$ if we don’t use the Reparametrization Trick:

• Replace $f$ with our $Q$ function here, and $z$ with our action $a$

In VPG, we don’t need this since we don’t actually sample new actions during the update, just the log probability of the action (see VPG implementation).

Implementation Detail

In VPG, TRPO, and PPO, we represent the log std devs with state-independent parameter vector (see VPG note where I also point this out). However, in SAC, we represent the log std devs as outputs from neural networks.

Why is that? State-dependent variance lets SAC adapt exploration locally rather than globally. This flexibility is critical for entropy regularization to balance exploration vs exploitation effectively.