Logistic Regression

machine-learning classification statistics

Definition

Logistic Regression

Logistic regression is a probabilistic linear classifier for binary classification that uses a linear score to parameterise a conditional probability for the class label.

Given an input $x\in {\mathbb{R}}^d$, a weight vector $w\in {\mathbb{R}}^d$, and a bias $b\in \mathbb{R}$, the model first computes

$$z=w^{\top }x+b.$$

It then maps this score through the sigmoid function and interprets the result as a conditional probability:

$$p(x)=P(Y=1\mid X=x)=\sigma (z)=\frac{1}{1+e^{-z}}.$$

Equivalently, $Y\mid X=x$ follows a Bernoulli distribution with parameter $p(x)$, so $P(Y=0\mid X=x)=1-p(x)$. The log-odds are linear,

$$\log \frac{p(x)}{1-p(x)}=w^{\top }x+b,$$

and the decision boundary at threshold $p(x)=\frac{1}{2}$ is the hyperplane $\{x\mid w^{\top }x+b=0\}$.

Optimisation

The parameters $w$ and $b$ are usually estimated by maximum likelihood estimation, equivalently by minimising the cross-entropy loss on labelled data $(x_i,y_i)$ with $y_i\in \{0,1\}$:

$$\mathcal{L}(w,b)=-\sum _{i=1}^m\left(y_i\log p(x_i)+(1-y_i)\log (1-p(x_i))\right).$$

For ordinary binary logistic regression, this objective is convex in $w$ and $b$ and is commonly optimised with gradient descent or related methods.