Experience Replay

From lilian-weng blog: https://lilianweng.github.io/posts/2018-02-19-rl-overview/#deep-q-network

It’s essentially storing the episode in the dataset, and sampling from it, as opposed to going through it sequentially. Seems analoguous to how we do Stochastic Gradient Descent in practice where we just shuffle a batch before using it as input training.

Experience Replay: All the episode steps $e_t=(S_t,A_t,R_t,S_{t+1})$ are stored in one replay memory $D_t=\{e_1,\dots ,e_t\}$. $D_t$ has experience tuples over many episodes. During Q-learning updates, samples are drawn at random from the replay memory and thus one sample could be used multiple times. Experience replay improves data efficiency, removes correlations in the observation sequences, and smooths over changes in the data distribution