Monte-Carlo Learning Model-Free Control

Monte-Carlo Control

Now, instead of the value function, we have the Q function that we want to determine.

GLIE Monte-Carlo Control

• Sample kth episode using $\pi :\{S_1,A_1,R_2,...,S_T\}\sim \pi$
• For each state $S_t$ and action $A_t$ in the episode, $N(S_t,A_t)\leftarrow N(S_t,A_t)+1$ $Q(S_t,A_t)\leftarrow Q(S_t,A_t)+\frac{1}{N(S_t,A_t)}(G_t-Q(S_t,A_t))$
• Improve policy based on new action-value function $ϵ\leftarrow 1/k$
  $\pi \leftarrow ϵ-greedy(Q)$

You need to understand the difference between On-Policy Methods and Off-Policy Methods. So far, the exploring starts is a solution to the On-Policy Methods method through Monte-Carlo.

The above updates seem so similar to Multi-Armed Bandit updates!