Probability Distribution

probability-theory measure-theory

Definition

Probability Distribution

Let $(\Omega ,\mathfrak{S},P)$ be a probability space, let $(\mathcal{X},\mathcal{B})$ be a measurable state space, and let

$$X:(\Omega ,\mathfrak{S})\to (\mathcal{X},\mathcal{B})$$

be a measurable random variable. The probability distribution or law of $X$ is the probability measure $P_X$ on $(\mathcal{X},\mathcal{B})$ defined by

$$P_X(B)=(X_{\ast }P)(B)=P(X^{-1}(B))=P(X\in B),B\in \mathcal{B}.$$

Thus $P_X$ assigns probability to measurable sets of values in $\mathcal{X}$. It is the pushforward of $P$ through $X$: outcomes $\omega \in \Omega$ receive probability under $P$, and $X$ transfers that probability to their values $X(\omega )\in \mathcal{X}$.

Equivalently, a probability distribution on a measurable space is any probability measure on that space. The notation $P_X$ emphasises that the measure is the distribution induced by $X$.

Why the Inverse Image Appears

The original measure $P$ accepts events in $\Omega$, not sets of values in $\mathcal{X}$. To find the probability that $X$ lands in a set $B\subseteq \mathcal{X}$, first pull $B$ back to the outcomes that produce such values:

$$B\stackrel{X^{-1}}{\to }\{\omega \in \Omega :X(\omega )\in B\}\stackrel{P}{\to }{P}_X(B).$$

Measurability of $X$ ensures that $X^{-1}(B)\in \mathfrak{S}$ whenever $B\in \mathcal{B}$, so $P$ is allowed to measure this event.

Why It Is a Probability Measure

The probability axioms pass from $P$ to $P_X$ through inverse images. In particular,

$$\begin{array}{rl}{P}_X(\mathcal{X})& =P(X^{-1}(\mathcal{X}))=P(\Omega )=1,\\ P_X(B)& \ge 0.\end{array}$$

If $B_1,B_2,\dots \in \mathcal{B}$ are pairwise disjoint, then their inverse images are also pairwise disjoint. Countable additivity therefore gives

$$\begin{array}{rl}{P}_X\left(\bigcup _{i=1}^{\infty }{B}_i\right)& =P\left(X^{-1}\left(\bigcup _{i=1}^{\infty }{B}_i\right)\right)\\ & =P\left(\bigcup _{i=1}^{\infty }{X}^{-1}(B_i)\right)\\ & =\sum _{i=1}^{\infty }P(X^{-1}(B_i))\\ & =\sum _{i=1}^{\infty }{P}_X(B_i).\end{array}$$

Representations

A distribution is the measure $P_X$ itself. Functions such as a PMF, PDF, or CDF are ways to represent that measure under additional structure.

Setting	Representation	Recovery of probabilities
Distribution supported on a countable set	PMF $p_X(x)=P_X(\{x\})$	$P_X(B)={\sum }_{x\in B}{p}_X(x)$
Distribution admitting a density	PDF $f_X$	$P_X(B)={\int }_B{f}_X(x)dx$
Real-valued distribution	CDF $F_X(t)=P_X((-\infty ,t])$	$F_X$ uniquely determines $P_X$

Warning

Not every distribution has a PMF or a PDF. Discrete distributions have PMFs, distributions absolutely continuous with respect to Lebesgue measure have PDFs, and mixed or singular distributions require the measure or CDF description. Every real-valued distribution has a CDF.

Example

Number of heads in two tosses

Let

$$\Omega =\{HH,HT,TH,TT\}$$

with each outcome having probability $1/4$, and let $X$ count the number of heads. Then

$$X(HH)=2,X(HT)=X(TH)=1,X(TT)=0.$$

The distribution of $X$ is

$$P_X(\{0\})=\frac{1}{4},P_X(\{1\})=\frac{1}{2},P_X(\{2\})=\frac{1}{4}.$$

For $B=\{1,2\}$, its inverse image is

$$X^{-1}(B)=\{HH,HT,TH\},$$

and hence

$$P_X(B)=P(X^{-1}(B))=\frac{3}{4}.$$

Joint Distributions

For a random vector $(X_1,\dots ,X_n)$ of binary variables, a general joint distribution assigns probabilities to $2^n$ configurations. Normalisation removes one degree of freedom, leaving $2^n-1$ free parameters.

If the variables are independent, the joint distribution factorises:

$$P(X_1=x_1,\dots ,X_n=x_n)=\prod _{i=1}^{n}P(X_i=x_i),$$

so only $n$ Bernoulli parameters are needed. Bayesian networks use conditional independence to obtain factorisations between these two extremes.