Linear Classifier

machine-learning classification

Definition

Linear Classifier

A linear classifier is a supervised learning model that makes classification decisions based on a linear combination of input features. Formally, for an input $\mathbf{x}\in {\mathbb{R}}^d$, the predicted class $y\in \{-1,+1\}$ is determined by a hyperplane in the instance space:

$$h(\mathbf{x})=\text{sign}({\mathbf{w}}^{\top }\mathbf{x}+b)$$

where $\mathbf{w}\in {\mathbb{R}}^d$ is the weight vector (normal to the hyperplane) and $b\in \mathbb{R}$ is the bias. Geometrically, the decision boundary is the set $\{\mathbf{x}:{\mathbf{w}}^{\top }\mathbf{x}+b=0\}$, which partitions the space into two half-spaces.

Training Strategies

Various algorithms exist to identify the optimal parameters $\mathbf{w}$ and $b$ depending on the objective function and data arrival paradigm.

Least Squares Classification: This approach approximates the discrete sign function with a linear identity $h(\mathbf{x})\approx {\mathbf{w}}^{\top }\mathbf{x}+b$ and solves the resulting system using the normal equations with target labels $y\in \{-1,+1\}$.

Perceptron Algorithm: An iterative online learning procedure that updates weights specifically based on misclassification errors, providing a hard decision boundary.

Logistic Regression: A probabilistic framework that maps the linear output to a confidence interval $(0,1)$ using the sigmoid function, allowing for uncertainty estimation.