Entropy

Maximum Entropy

This is actually super useful and practical, because the world is full of Uncertainty.

The Entropy is given by $H(X)=-{\sum }_{x\in X}p(x)\log p(x)$

This is another way to formulate. To take into account uncertainty, so for Robustness.

$V={\max }_{\pi (a)}(E[r(a)]+\beta H(\pi (a)))$

We use Constrained Optimization to come up with a set of equations.

This is a really important derivation

ahh you want to maximize entropy so that the policy is not as deterministic

Max-entropy Value Iteration $V_k(s)={\max }_{\pi }(\mathbb{E}[R(s,a)+V_{k-1}(s^{\prime })]+\beta H(\pi (a|s)))$