# Markov Chain Monte-Carlo

Markov chain Monte Carlo (MCMC) is a technique for estimating by simulation the expectation of a statistic in a complex model.

Two good blog articles:

- https://jeremykun.com/2015/04/06/markov-chain-monte-carlo-without-all-the-bullshit/
- https://machinelearningmastery.com/markov-chain-monte-carlo-for-probability/

This is one of the most influential ideas of the 20th century.

https://www.youtube.com/watch?v=12eZWG0Z5gY&t=12s&ab_channel=mathematicalmonk Goal: Sample from $p$, or approximate $Ef(x)(X∼p)$

I found this 50 page explanation: https://arxiv.org/pdf/1909.12313.pdf

This is a good blog: https://jeremykun.com/2015/04/06/markov-chain-monte-carlo-without-all-the-bullshit/ We can use MCMC to approximate some some sort of unknown distribution, by modelling it as some sort of Random Walk on a Graph, where we can apply the Metropolis-Hastings Algorithm?

How is this different from purely monte-carlo method?

Referenced from the original article for Reinforcement Learning: https://lilianweng.github.io/posts/2018-02-19-rl-overview/ Referenced from the original article for Reinforcement Learning:

Mentioned also in the original GAN paper.

## Flavors of MCMC

### Replica-Exchange MCMC / Parallel Tempering

This “is a standard approach to maintaining a population of configurations” from FormulaZero.

### Hit-and-Run Algorithm

Hit-and-run is a Markov chain Monte Carlo (MCMC) sampling technique that iteratively generates a sequence of points in a set by taking steps of random length in randomly generated directions. Hit-and-run can be applied to virtually any bounded region in {\Re^n} , and has nice convergence properties.Jan 23, 2016