Naive Bayes

machine-learning classification statistics

Definition

Naive Bayes

Naive Bayes is a probabilistic generative classifier based on Bayes’ theorem with the “naive” assumption of conditional independence between features given the class label. Formally, for an instance $\mathbf{x}=(x_1,\dots ,x_d)$ and class $y$, the joint probability is modelled as:

$P(\mathbf{x},y)=P(y){\prod }_{j=1}^{d}P(x_j|y)$

Inference and Training

Classification Rule: The predicted label is identified by maximising the numerator of the posterior (since the evidence $P(\mathbf{x})$ is constant for all classes):
$\hat{y}=\arg {\max }_{y\in \mathcal{Y}}P(y){\prod }_{j=1}^{d}P(x_j|y)$

Parameter Estimation: The class priors $P(y)$ and class-conditional probabilities $P(x_j|y)$ are typically estimated via MLE (relative frequencies) or MAP (to avoid zero probabilities through smoothing).