Linear Regression

statistics regression machine-learning

Definition

Linear Regression

Linear regression models the relationship between a dependent variable $y\in \mathbb{R}$ and one or more independent variables $\mathbf{x}\in {\mathbb{R}}^d$ by assuming a linear dependency. Formally, for an observation $({\mathbf{x}}_i,y_i)$, the model assumes:

$y_i={\mathbf{w}}^{\top }{\mathbf{x}}_i+b+{ϵ}_i$

where $\mathbf{w}$ is the coefficient vector (slope), $b$ is the intercept (bias), and ${ϵ}_i\sim \mathcal{N}(0,{\sigma }^2)$ represents normally distributed error or noise. The objective is to identify the hyperplane that best fits the observed data by minimising the discrepancy between the predicted values ${\hat{y}}_i$ and actual observations $y_i$.

Estimation: Minimising Squared Error

The parameters $\mathbf{w}$ and $b$ are typically estimated by minimising the Mean Squared Error (MSE), which represents the average squared residual over $n$ samples:

$J(\mathbf{w},b)=\frac{1}{n}{\sum }_{i=1}^n(y_i-({\mathbf{w}}^{\top }{\mathbf{x}}_i+b){)}^2$

Normal Equations (Exact Solution): When the feature dimension $d$ is manageable, the optimal parameters can be calculated analytically. By prepending a constant $1$ to $\mathbf{x}$ to absorb $b$ into $\mathbf{w}$, the solution is $\mathbf{w}=(X{X}^{\top }{)}^{-1}X\mathbf{y}$. This provides the global minimum in a single step without requiring hyperparameters.

Gradient Descent (Iterative Optimisation): For high-dimensional datasets or online learning settings, parameters are updated iteratively in the direction of the negative gradient: $\theta \leftarrow \theta -\eta \nabla J(\theta )$. This approach scales better to large data but requires the selection of a learning rate $\eta$.