Orthogonality Splits Space

An inner product space gives us a way to say that two directions do not overlap. That is what orthogonality means:

$$⟨u,w⟩=0.$$

This one equation is the source of orthogonal complements, orthogonal projections, and the clean part of spectral representation. Orthogonality lets a vector be split into independent pieces.

Perpendicular means no shared component

Let $U$ be a subspace of an inner product space $V$. The orthogonal complement $U^{\perp }$ is the set of all vectors perpendicular to every vector in $U$:

$$U^{\perp }=\{w\in V:⟨w,u⟩=0\text{ for all }u\in U\}.$$

So $U^{\perp }$ is not merely “another direction”. It is the collection of directions that have no component inside $U$.

In finite dimensions, the cleanest case is

$$V=U\oplus U^{\perp }=\{u+w\mid u\in U,w\in U^{\perp }\}.$$

Each subspace stays separate: a vector in $U$ cannot be built from vectors in $U^{\perp }$, except for the zero vector. Instead, the whole space is rebuilt by adding one component from each side.

The symbol $\oplus$ also records uniqueness: every vector $x\in V$ has exactly one decomposition

$$x=u+w,u\in U,w\in U^{\perp }.$$

The two pieces do not interfere. One lives in $U$; the other is perpendicular to all of $U$.

Projection chooses the kept part

An orthogonal projection onto $U$ is the operator that keeps the $U$-part and removes the perpendicular part:

$$P_U(u+w)=u.$$

Here $u\in U$ and $w\in U^{\perp }$. The vector $u$ is the shadow of $x$ on $U$; the vector $w=x-u$ is the leftover error, and it is perpendicular to $U$.

So projection is not “dropping a vector down” by convention. It is the unique split

$$x=\underset{\text{kept in }U}{\underbrace{P_{U}x}}+\underset{\text{removed in }{U}^{\perp }}{\underbrace{(x-P_{U}x)}}.$$

Orthogonality tells us which leftover is legitimate: the leftover must have no component in $U$.

The closest point condition

The projection $P_{U}x$ is also the closest point in $U$ to $x$. The reason is geometric but algebraic.

Take any other $y\in U$. Since $P_{U}x\in U$, the difference $P_{U}x-y$ lies in $U$. The residual $x-P_{U}x$ lies in $U^{\perp }$. Therefore the two pieces are orthogonal:

$$⟨x-P_{U}x,P_{U}x-y⟩=0.$$

Then the squared distance from $x$ to $y$ splits by Pythagoras:

$$\begin{array}{rl}\Vert x-y{\Vert }^2& =\Vert (x-P_{U}x)+(P_{U}x-y){\Vert }^2\\ & =\Vert x-P_{U}x{\Vert }^2+\Vert P_{U}x-y{\Vert }^2.\end{array}$$

The second term is non-negative, so the smallest distance occurs at $y=P_{U}x$.

This is why the projected point is not arbitrary. It is the best approximation to $x$ inside $U$.

Operator view: kept means eigenvalue 1, removed means eigenvalue 0

The projection is a linear operator. Its behaviour is completely simple on the two perpendicular parts:

$$P_{U}u=1\cdot u=u(u\in U),P_{U}w=0\cdot w=0(w\in U^{\perp }).$$

So $U$ is the eigenspace for eigenvalue $1$, and $U^{\perp }$ is the eigenspace for eigenvalue $0$:

$$U=E_1(P_U),U^{\perp }=E_0(P_U).$$

This is the spectral meaning of projection. Projection separates the space into two perpendicular eigenspaces and applies the scalars $1$ and $0$.

Spectral representation is many projections at once

An orthogonal projection is the smallest spectral picture:

$$V=U\oplus U^{\perp },P_U=1\cdot P_U+0\cdot P_{U^{\perp }}.$$

A spectral representation generalises this idea. Instead of only two parts, a nice operator splits the space into mutually orthogonal eigenspaces:

$$V=E_{{\lambda }_1}\oplus E_{{\lambda }_2}\oplus \cdots \oplus E_{{\lambda }_k}.$$

On each part, the operator is just scaling:

$$Tx={\lambda }_{i}x\text{for }x\in E_{{\lambda }_i}.$$

If $P_i$ denotes the orthogonal projection onto $E_{{\lambda }_i}$, then

$$T={\lambda }_1{P}_1+{\lambda }_2{P}_2+\cdots +{\lambda }_k{P}_k.$$

The projections separate the vector into orthogonal spectral components. The eigenvalues say how much each component is scaled.

So orthogonal projection is the training-wheel version of spectral representation. It teaches the core move: split the vector into perpendicular components, then act on each component independently.