A quadratic boundary can represent shapes that a linear boundary cannot. It is tempting to choose the richer family whenever the true boundary might be curved.
But the boundary is not given to us. We must estimate it from observations, and each extra coefficient is another quantity the data must determine.
Count the terms being fitted
In two dimensions, write the two scores explicitly:
The quadratic score has three additional coefficients, one for each second-degree term. The retained linear and constant terms occupy the same positions below.
In
One scale does not change the classifier
For any
For a non-zero coefficient vector, fixing its scale removes one continuous degree of freedom. This gives the generic counts
In the diagram, the corresponding counts are
These counts concern general polynomial boundary scores, not every parameter of a generative Gaussian model. A full shared covariance matrix still has
The best known rule need not be the best estimated rule
With known Gaussian parameters and unequal class covariances, the Bayes rule is generally quadratic. With limited data, however, estimates of the many coefficients may be unstable.
Restricting the model to linear boundaries can increase approximation error while reducing estimation error. The linear model can therefore predict better from a small training set, even when it cannot express the true Bayes boundary.
Parameter counts explain a source of estimation difficulty; they do not establish a universal minimum sample size. Reliability also depends on covariance conditioning, noise, separation and regularisation.
The relevant comparison is not only which boundary fits the population, but which boundary the available data can estimate reliably.