Logistic Regression

machine-learning classification statistics

Definition

Logistic Regression

Logistic regression is a probabilistic linear classifier that estimates the probability that an instance $\mathbf{x}\in {\mathbb{R}}^d$ belongs to a specific class. Formally, the model applies the sigmoid function to a linear combination of features:

$$P(y=1|\mathbf{x})=\sigma ({\mathbf{w}}^{\top }\mathbf{x}+b)=\frac{1}{1+\exp (-({\mathbf{w}}^{\top }\mathbf{x}+b))}$$

where $\mathbf{w}$ is the weight vector and $b$ is the bias. Unlike the perceptron, logistic regression provides a continuous confidence estimate $p\in [0,1]$ rather than a discrete label.

Optimisation

The parameters $\mathbf{w}$ and $b$ are identified by minimising the empirical risk with the log-loss (cross-entropy) function:

$$R_{\text{emp}}(\mathbf{w})=\sum _{i=1}^m\log (1+\exp (-y_i{\mathbf{w}}^{\top }{\mathbf{x}}_i))$$

where $y_i\in \{-1,+1\}$. Since the objective is convex, it is typically solved using gradient descent.