Variance

statistics

Definition

Variance

Variance measures how far a set of numbers is spread out from their mean. It is the average of the squared differences from the mean. A high variance indicates a large spread of the numbers in the dataset.

$${\sigma }^2=\frac{1}{n}\sum _{i=1}^n{\left({\mathbf{x}}_i-{\mu }_{\mathbf{x}}\right)}^2$$

where $n$ is the number of elements in the population $\mathbf{x}$ and ${\mu }_{\mathbf{x}}$ is the mean of the population.

Variance

Variance $\mathbb{V}ar(X)$ is a measure of dispersion of a random variable $X$ and is given by:

$$\mathbb{V}ar(X)=\mathbb{E}[{\left(X-\mathbb{E}[X]\right)}^2]=\mathbb{E}[X^2]-(\mathbb{E}[X]{)}^2$$

Properties

Let $X,Y$ be two random variables and $a,b\in \mathbb{R}$ be scalars. Then:

Linearity

The variance is linear for any real-valued random variables $X,Y$ and any real-valued scalar $a\in \mathbb{R}$:

$$\mathbb{V}ar(A\cdot X+b)=a^2\cdot \mathbb{V}ar(X)$$

Sum

$$\mathbb{V}ar(X+Y)=\mathbb{V}ar(X)+\mathbb{V}ar(Y)+2\cdot \mathbb{C}ov(X,Y)$$

Note that if $X$ and $Y$ are independent, then $\mathbb{C}ov(X,Y)=0$ and:

$$\mathbb{V}ar(X+Y)=\mathbb{V}ar(X)+\mathbb{V}ar(Y)\forall X,Y:X\perp Y$$

Geometric Analogy

This equation remembers me of the cosine law from trigonometry.

The formula for the variance of a sum of random variables is the statistical equivalent of the geometric Law of Cosines. This reveals an analogy where random variables can be treated as vectors in an abstract space.

The direct correspondence is as follows:

Geometric Concept (Vectors)	Statistical Concept (Random Variables)
Vector, $v$	Random Variable, $X$
Squared Length, $\Vert v{\Vert }^2$	Variance, $\mathbb{V}ar(X)$
Dot Product, $a\cdot b$	Covariance, $\mathbb{C}ov(X,Y)$
Cosine of Angle, $\cos (\theta )$	Correlation, $\rho (X,Y)$

The Law of Cosines for the sum of two vectors $a$ and $b$ is:

$$\Vert a+b{\Vert }^2=\Vert a{\Vert }^2+\Vert b{\Vert }^2+2(a\cdot b)$$

This perfectly mirrors the variance formula:

$$\mathbb{V}ar(X+Y)=\mathbb{V}ar(X)+\mathbb{V}ar(Y)+2\cdot \mathbb{C}ov(X,Y)$$

In this analogy, the correlation coefficient ($\rho$) is literally the cosine of the angle between the two random variable vectors, making the connection mathematically precise.

The connection gets even deeper: The dot product in geometry is defined as:

$$a\cdot b=||a||\cdot ||b||\cdot \cos (\theta )$$

In probability theory, the covariance is related to correlation by:

$$\mathbb{C}ov(X,Y)={\sigma }_X\cdot {\sigma }_Y\cdot \rho (X,Y)$$

Intuition

Intuition

After finding the centre, the next logical question is: “How spread out are the data points from this centre?“. Are they all clustered tightly around the mean, or are they widely dispersed.

To measure this, one could try calculating the average distance of each point from the mean, i.e.:

$$\mathbb{E}[X-\mathbb{E}[X]]$$

However, some points are to the left mean (negative distance) and some are to the right (positive distance). These will cancel each other out, and the result will always be zero.

Thus, all distances should be non-negative. There are two obvious main options:

1. Absolute value: $\mathbb{E}[\left|X-\mathbb{E}[X]\right|]$
2. Squaring: $\mathbb{E}[{\left(X-\mathbb{E}[X]\right)}^2]$

Statisticians prefer squaring for a few reasons: it’s mathematically easier to work with (continuous, differentiable), and it heavily penalises values that are far from the mean, which is often a desirable property.

This measure, the average of the squared distances from the mean, is exactly the variance:

$$\begin{array}{rl}\mathbb{V}ar(X)=\mathbb{E}[(X-\mathbb{E}[X]{)}^2]& =\text{2nd Central Moment}\\ & =\text{Spread or Dispersion}\end{array}$$Link to original