Distribution

Distance between Distributions

How do you quantify how far away two distributions are? There seems to be quite a few methods, that I found from stackoverflow

I ran into this problem while working on my Poker AI, since Euclidean distance is not the best measure.

Heuristic

• Minkowski-form
• Weighted-Mean-Variance (WMV)

Nonparametric test statistics

• 2 (Chi Square)
• Kolmogorov-Smirnov (KS)
• Cramer/von Mises (CvM)

Information-theory divergences

• Cross-Entropy
• KL Divergence
• Jensen–Shannon divergence (metric)
• Jeffrey-divergence (numerically stable and symmetric)

Note that for cross-entropy and kl divergence, they are not true measures, since $D_{KL}(P||Q)\ne D_{KL}(Q||P)$

Ground distance measures

• Histogram intersection
• Quadratic form (QF)
• Earth Mover’s Distance