Markov Decision Process

Partially Observable Markov Decision Process

A Partially Observable Markov Decision Process is an MDP with hidden states. It is a hidden Markov model with actions.

• i.e. When the environment is only partially observable by the agent, ex: game of poker.

Highlighted in red is what we add to the original MDP.

Definition

A POMDP is a tuple $⟨\mathcal{S},\mathcal{A},\mathcal{O},\mathcal{P},\mathcal{R},\mathcal{Z},\gamma ⟩$

• $\mathcal{S}$ is a finite set of states
• $\mathcal{A}$ is a finite set of actions
• $\mathcal{O}$ is a finite set of observations
• $\mathcal{P}$ is a state transition probability matrix
• ${\mathcal{P}}_{s{s}^{\prime }}^a=\mathbb{P}[S_{t+1}=s^{\prime }\mid S_t=s,A_t=a]$
• $\mathcal{R}$ is a reward function, $R_s^a=\mathbb{E}[R_{t+1}\mid S_t=s,A_t=a]$
• $\mathcal{Z}$ is an observation function
• ${\mathcal{Z}}_{s^{\prime }o}^a=\mathbb{P}[O_{t+1}=o\mid S_{t+1}=s^{\prime },A_t=a]$
• $\gamma$ is a discount factor, $\gamma \in [0,1]$