Variance

Variance is Standard Deviation squared, i.e. $σ^{2}$ . $σ^{2} = Va r (X) = E [X - E (X)]^{2} = E (X^{2}) - [E (X)]^{2} = E (X^{2}) - μ^{2}$ I just realized that this is kind of the formula for Squared Euclidean Distance, and this was talked about the Curse of Dimensionality, which is $σ^{2} \to 0$ as the dimensionality increases.

From the Standard Deviation formula, I guess we can say the variance of a sample can also be expressed as $σ^{2}$ represents the variance of a population, $σ^{2} = \frac{1}{n} \sum_{i = 1}^{n} (x_{i} - \overset{x}{ˉ})^{2}$

If you are looking the variance of a sample, this would be given by $s^{2} = \frac{1}{n - 1} \sum_{i = 1}^{n} (x_{i} - \overset{x}{ˉ})^{2} = \frac{1}{n - 1} (\sum x_{i}^{2} - \frac{( \sum x _{i} ) ^{2}}{n})$

I think variance of a Distribution vs. variance of a sample is very different. I don’t know how the later is derived.

$Va r (x) = E (x^{2}) - (E (x))^{2}$
$Va r (x) = \int_{- \infty}^{\infty} x^{2} f (x) d x - [\int_{- \infty}^{\infty} x (f (x)]^{2}$ “average of squares” and the square of sums

Properties:

If $X$ is a constant, then $Va r (X) = 0$
Standard Deviation $σ = Va r (X)$
$Va r (a X) = a^{2} Va r (X)$
$Va r (a X + b) = a^{2} Va r (X)$
$Va r (X + Y) = Va r (X) + Va r (Y) + 2 C o v (X, Y)$ see Covariance

Incremental Variance

Learning about this as I am on Ericsson’s team for an AI study. The problem is that when we have memory constraints, we cannot store all of it and then compute it afterwards, using the standard equation.

We want a recurrence relation. https://math.stackexchange.com/questions/374881/recursive-formula-for-variance

https://math.stackexchange.com/questions/2148877/iterative-calculation-of-mean-and-standard-deviation

https://en.wikipedia.org/wiki/Algorithms_for_calculating_variance

I used Welford’s Online Algorithm, just check out the wikipedia article.