Multiclass Classification

machine-learning classification statistics

Definition

Multiclass Classification

Multiclass classification is a supervised learning task where the objective is to categorise instances into one of more than two distinct classes. Formally, given an instance space $\mathcal{X}$, the learner seeks a mapping $h:\mathcal{X}\to \mathcal{Y}$ where the label space $\mathcal{Y}=\{y_1,y_2,\dots ,y_k\}$ has cardinality $k>2$.

Reduction Strategies

Since many foundational algorithms (e.g., SVM, Perceptron) are inherently binary, multiclass problems are often decomposed into multiple binary sub-problems.

One-vs-Rest (OvR): Also known as one-vs-all, this strategy trains $k$ separate binary classifiers, each designed to distinguish one class from all other classes combined. The final prediction is the class whose classifier produces the highest confidence score.

One-vs-One (OvO): This strategy trains $k(k-1)/2$ binary classifiers, one for every possible pair of classes. The final label is determined by a majority vote among all pairs.

Softmax Regression: A direct multiclass extension of logistic regression that utilises the softmax function to produce a probability distribution over all $k$ classes, typically trained using categorical cross-entropy.