Linear Discriminant Analysis

Definition

Linear Discriminant Analysis

Linear discriminant analysis (LDA), as a classifier, models classes by multivariate Gaussian distributions with a shared positive-definite covariance matrix . For two classes,

Its class scores and prediction are

A fixed convention resolves ties. With the true parameters, this is the Bayes optimal classifier under zero–one loss. For distinct means, the boundary between the classes is an affine hyperplane.

Why the quadratic terms cancel

In the Gaussian log-density scores, each class contributes

The term is also common. Removing both common terms leaves the scores in the definition. Their difference is

Thus selects class , and selects class . Equal priors remove the log-prior term. If , the score is constant and there is no non-trivial separating hyperplane.

Estimating the shared covariance

For observations in class , let and be its sample mean and unbiased sample covariance. The pooled estimate is

Using a shared covariance restricts the model even if the actual class covariances differ. This can reduce estimation error, but the resulting classifier is no longer guaranteed to be Bayes optimal. If the estimate is singular, inversion requires a remedy such as regularisation or dimensionality reduction.