Bayes Optimal Classifier

machine-learning statistics learning-theory

Definition

Bayes Optimal Classifier

The Bayes optimal classifier is the theoretically best possible classification model for a given data distribution . Formally, for an input , the Bayes optimal prediction is obtained by selecting the label that maximises the posterior probability:

This classifier achieves the minimum possible true risk, known as the Bayes risk. For any other hypothesis , the risk holds.

Optimality and Risk

Bayes Risk: The expected error rate of the Bayes optimal classifier. It represents the irreducible error caused by the overlap of class distributions in the feature space (stochasticity).

Relation to Empirical Methods: While the true distribution is typically unknown, machine learning algorithms seek to approximate the Bayes optimal classifier by either directly estimating the conditional distribution (Discriminative Learning) or modelling the joint distribution (Generative Learning).