Polynomial Regression

machine-learning regression statistics

Definition

Polynomial Regression

Polynomial regression is a generalisation of linear regression where the relationship between the independent variable $x$ and the dependent variable $y$ is modelled as an $n$-th degree polynomial. Although the relationship is non-linear, the model remains linear in its parameters $\theta$.

Formally, the hypothesis $h_{\theta }(x)$ is:
$h_{\theta }(x)={\sum }_{j=0}^n{\theta }_j{x}^j={\theta }^{\top }\varphi (x)$
where $\varphi (x)=[1,x,x^2,\dots ,x^n{]}^{\top }$ is the feature transformation.

…

% Axes
\draw[→, cyan!30!white] (0,0) — (2.8, 0) node[right, text=cyan!30!white, font=\bfseries] {$x$};
\draw[→, cyan!30!white] (0,0) — (0, 2.2) node[above, text=cyan!30!white, font=\bfseries] {$y$};
\end{tikzpicture}
\end{document}

Model Complexity

The degree $n$ of the polynomial determines the capacity of the learner. Lower degrees may result in underfitting (high bias), while excessively high degrees lead to overfitting (high variance) by capturing noise in the training data.