Policy Evaluation

Problem: Evaluate a given policy $\pi$ Solution: Iterative application of Bellman Expectation Backup

• $v_1\to v_2\to ...\to v_{\infty }=v_{\pi }$
• Using synchronous backups,
  • At each iteration, for all states $s\in S$ Update $v_{k+1}(s)$ from $v_k(s^{\prime })$, where $s^{\prime }$ is a successor state of $s$ $v_{k+1}(s)={\sum }_a\pi (a|s){\sum }_{s^{\prime },r}p(s^{\prime },r|s,a)(r+\gamma v_k(s^{\prime }))$
• Make sure to know this equation by heart (bellman expectation backup), and the difference with the bellman optimality equation
• You should also know for the deterministic Value Function

Pseudocode

Question: from the stanford thing, they tell us to initialize $v(s)$ to 0, but here its random except the terminal value function is also 0. Is it just for faster convergence?#gap-in-knowledge

Related

• Policy Improvement
• Generalized Policy Iteration