Dimensionality Reduction

machine-learning

Dimensionality Reduction

Dimensionality reduction is the process of transforming data from a high-dimensional space into a low-dimensional space while preserving its essential structure and meaningful properties. The primary goal is to obtain a more compact data representation to facilitate visualization, improve computational efficiency, and mitigate the Curse of Dimensionality.

Formalism

Let a dataset be represented by points $x_i\in {\mathbb{R}}^D$ in a high-dimensional space. Dimensionality reduction seeks a mapping function $f:{\mathbb{R}}^D\to {\mathbb{R}}^d$, where the new dimension $d$ is significantly smaller than the original dimension $D$ (i.e., $d\ll D$). This function transforms each data point $x_i$ into a new representation $z_i=f(x_i)$ in the lower-dimensional space.

Main Approaches

1. Feature Projection: Creates new, lower-dimensional features by computing combinations of the original ones (e.g., PCA).
2. Feature Selection: Identifies and retains a subset of the most relevant original features, discarding the rest.