Covariance

probability-theory

Definition

Covariance

The covariance $\mathbb{C}ov(X,Y)$ between two random variables is a measure of alignment and joint variability, and is defined as:

$$\mathbb{C}ov(X,Y)=\mathbb{E}[(X-\mathbb{E}[X])(Y-\mathbb{E}[Y])]$$

Properties

Expanded Form

$$\mathbb{C}ov(X,Y)=\mathbb{E}[X\cdot Y]-\mathbb{E}[X]\cdot \mathbb{E}[Y]$$

Symmetry

The covariance is symmetric for any real-valued random variables $X,Y$:

$$\mathbb{C}ov(X,Y)=\mathbb{C}ov(Y,X)$$

Affine Linearity

The covariance is affine linear for any real-valued random variables $X,Y$ and real-valued scalar $a\in \mathbb{R}$:

$$\mathbb{C}ov(a\cdot X+b,Y)=a\cdot \mathbb{C}ov(X,Y)$$

Bilinear Form

The covariance is a bilinear form in $(X,Y)$ for any real-valued random variables $X,Y,Z$:

$$\mathbb{C}ov(X+Y,Z)=\mathbb{C}ov(X,Z)+\mathbb{C}ov(Y,Z)$$

(i.e. additivity in the first argument)

Independence

If two random variables $X,Y$ are independent, i.e. $X\perp Y$, then:

$$X\perp Y⟹\mathbb{C}ov(X,Y)=0$$

The reverse, $\mathbb{C}ov(X,Y)=0⟹X\perp Y$, is generally wrong.

Example

For example, consider $X\sim U(-1,1)$ and $Y=X^2$. Since $Y$ is a (non-trivial) function of $X$, the two random variables are not independent, but:

$$\begin{array}{r}\mathbb{C}ov(X,Y)=\mathbb{E}[X\cdot Y]-\mathbb{E}[X]\cdot \mathbb{E}[Y]=\mathbb{E}[X^3]-0\cdot \mathbb{E}[Y]=\frac{1}{2}{\int }_{-1}^1{x}^{3}dx=0\end{array}$$

Therefore, $\mathbb{C}ov(X,Y)=0/⟹X\perp Y$.

Linear Dependence

The product term $\mathbb{C}ov(X,Y)=(X-\mathbb{E}[X])(Y-\mathbb{E}[Y])$ tells us whether $X$ and $Y$ are on the same side of their respective means.

We can analyse five possible scenarios for any given data point $(x,y)$:

1. $X>\mathbb{E}[X]\wedge Y>\mathbb{E}[Y]$
   • $⟹(X-\mathbb{E}[X])>0$
   • $⟹(Y-\mathbb{E}[Y])>0$
   • $⟹(X-\mathbb{E}[X])(Y-\mathbb{E}[Y])>0$
2. $X<\mathbb{E}[X]\wedge Y<\mathbb{E}[Y]$
   • $⟹(X-\mathbb{E}[X])<0$
   • $⟹(Y-\mathbb{E}[Y])<0$
   • $⟹(X-\mathbb{E}[X])(Y-\mathbb{E}[Y])>0$
3. $X>\mathbb{E}[X]\wedge Y<\mathbb{E}[Y]$
   • $⟹(X-\mathbb{E}[X])>0$
   • $⟹(Y-\mathbb{E}[Y])<0$
   • $⟹(X-\mathbb{E}[X])(Y-\mathbb{E}[Y])<0$
4. $X<\mathbb{E}[X]\wedge Y>\mathbb{E}[Y]$
   • $⟹(X-\mathbb{E}[X])<0$
   • $⟹(Y-\mathbb{E}[Y])>0$
   • $⟹(X-\mathbb{E}[X])(Y-\mathbb{E}[Y])<0$
5. $X=\mathbb{E}[X]\vee Y=\mathbb{E}[Y]$
   • $⟹(X-\mathbb{E}[X])(Y-\mathbb{E}[Y])=0$

The covariance $\mathbb{E}[(X-\mathbb{E}[X])(Y-\mathbb{E}[Y])]$ is the expectation of this product over all possible outcomes.

1. Case: strong positive linear dependence:
   • If $X$ and $Y$ are positively related, they tend to move together. This means scenarios 1 and 2 (above, where the product is positive) will happen much more often than scenarios 3 and 4. When taking the expectation, the many positive terms will overpower the few negative terms.
   • Result: $\mathbb{C}ov(X,Y)$ will be a large positive number.
2. Case: strong negative linear dependence:
   • If $X$ and $Y$ are negatively related, they tend to move in opposite directions. This means scenarios 4 and 4 (above, where the product is negative) will happen much more often then scenarios 1 and 2. When taking the expectation, the many negative terms will overpower the few positive terms.
   • Result: $\mathbb{C}ov(X,Y)$ will be a large negative number.
3. Case 3: no linear dependence:
   • If $X$ and $Y$ have no linear relationship, all scenarios 1-4 (above) are roughly balanced. The positive products from scenarios 1 and 2 will be cancelled out, on average, by the negative products from scenarios 3 and 4 (and vice versa).
   • Result: $\mathbb{C}ov(X,Y)$ will tend to zero.

The covariance $\mathbb{C}ov(X,Y)$ measures the linear dependence between $X$ and $Y$.