Linear Regression

statistics regression machine-learning

Definition

Linear Regression

Linear regression models the relationship between a dependent variable and one or more independent variables by assuming a linear dependency. Formally, for an observation , the model assumes:

where is the coefficient vector (slope), is the intercept (bias), and represents normally distributed error or noise. The objective is to identify the hyperplane that best fits the observed data by minimising the discrepancy between the predicted values and actual observations .

Estimation: Minimising Squared Error

The parameters and are typically estimated by minimising the Mean Squared Error (MSE), which represents the average squared residual over samples:

Normal Equations (Exact Solution): When the feature dimension is manageable, the optimal parameters can be calculated analytically. By prepending a constant to to absorb into , the solution is . This provides the global minimum in a single step without requiring hyperparameters.

Gradient Descent (Iterative Optimisation): For high-dimensional datasets or online learning settings, parameters are updated iteratively in the direction of the negative gradient: . This approach scales better to large data but requires the selection of a learning rate .