Supervised Learning

machine-learning supervised-learning

Definition

Supervised Learning

Supervised learning is a paradigm of machine learning where the objective is to learn a mapping from an instance space $\mathcal{X}$ to a label space $\mathcal{Y}$ based on provided input-output pairs. Formally, given a training dataset $\mathcal{D}=\{(x_i,y_i){\}}_{i=1}^n\subset \mathcal{X}\times \mathcal{Y}$ sampled i.i.d. from a joint distribution $P(X,Y)$, the learner aims to identify a hypothesis $h\in \mathcal{H}$ that minimises the expected risk $R(h)={\mathbb{E}}_{(x,y)\sim P}[L(h(x),y)]$ for a given loss function $L$.

Data Representation

The supervised setting assumes a structural relationship between input and output, defined by the geometry of the data and the capacity of the learner. The learning process operates on an instance space $\mathcal{X}$ (typically ${\mathbb{R}}^d$) and a label space $\mathcal{Y}$. The nature of $\mathcal{Y}$ distinguishes between tasks: discrete labels define classification, while continuous values define regression.

Inductive Bias

To prevent an unconstrained search, the learner is restricted to a hypothesis class $\mathcal{H}\subseteq \{h:\mathcal{X}\to \mathcal{Y}\}$. This restriction encodes the inductive bias, allowing the model to generalise from finite samples to the full distribution.

Empirical Risk Minimisation

Since the true distribution $P(X,Y)$ is unknown, the model is trained by identifying a hypothesis $h\in \mathcal{H}$ that minimises the loss over the available data. This principle, known as Empirical Risk Minimisation (ERM), seeks to approximate the optimal mapping by penalising discrepancies between the predicted and true labels.

Definition

Supervised Learning Algorithm

A supervised learning algorithm $A$ is a learning algorithm that maps a set of training data to a specific hypothesis:

$$A:(\mathcal{X}\times \mathcal{Y}{)}^m\to \mathcal{H}$$

where:

• $\mathcal{X}$ is the instance space.
• $\mathcal{Y}$ is the label space.
• $\mathcal{H}$ is the hypothesis class.

Link to original