Bayes Optimal Classifier

machine-learning statistics learning-theory

Definition

Bayes Optimal Classifier

The Bayes optimal classifier is the theoretically best possible classification model for a given data distribution $P(X,Y)$. Formally, for an input $\mathbf{x}\in \mathcal{X}$, the Bayes optimal prediction $f^{\ast }(\mathbf{x})$ is obtained by selecting the label $y\in \mathcal{Y}$ that maximises the posterior probability:

$f^{\ast }(\mathbf{x})=\arg {\max }_{y\in \mathcal{Y}}P(y|\mathbf{x})$

This classifier achieves the minimum possible true risk, known as the Bayes risk. For any other hypothesis $g:\mathcal{X}\to \mathcal{Y}$, the risk $R(f^{\ast })\le R(g)$ holds.

Optimality and Risk

Bayes Risk: The expected error rate of the Bayes optimal classifier. It represents the irreducible error caused by the overlap of class distributions in the feature space (stochasticity).

Relation to Empirical Methods: While the true distribution $P(y|\mathbf{x})$ is typically unknown, machine learning algorithms seek to approximate the Bayes optimal classifier by either directly estimating the conditional distribution (Discriminative Learning) or modelling the joint distribution (Generative Learning).