Markov Chain

statistics

Definition

Markov Chain

A Markov chain is a discrete-time stochastic process $(X_t{)}_{t\ge 0}$ that satisfies the Markov property: the conditional distribution of the next state depends only on the current state, not on the full history.

Its evolution is fully specified by:

• A countable or finite state space $\mathcal{S}$.
• Transition probabilities $P(s,s^{\prime })=\mathbb{P}(X_{t+1}=s^{\prime }\mid X_t=s)$ for all $s,s^{\prime }\in \mathcal{S}$.
• An initial distribution ${\pi }_0(s)=\mathbb{P}(X_0=s)$.

The transition probabilities are often collected in a transition matrix $\mathbf{P}$ with entries $P_{ij}=P(i,j)$.

Properties

Time-homogeneity

A Markov chain is time-homogeneous if the transition probabilities do not change over time: $P(X_{t+1}=s^{\prime }\mid X_t=s)$ is the same for all $t$. Otherwise it is time-inhomogeneous.

Stationary distribution

A probability distribution $\pi$ over $\mathcal{S}$ is stationary if it satisfies $\pi =\pi \mathbf{P}$. Under certain conditions (irreducibility and aperiodicity), the chain converges to its unique stationary distribution regardless of the initial state.