Dot Product

linear-algebra

Definition

Dot Product

Let $x,y\in V$ be two vectors of vector space $V\cong {\mathbb{F}}^d$, $\mathbb{F}$ being a scalar field.

The dot product $⟨\cdot ,\cdot ⟩:V\times V\to \mathbb{F}$ is an inner product defined as the sum over the dimension $d$

$$\begin{array}{rl}⟨x,y⟩& =x\cdot y\\ & =xy\\ & :=\sum _{i=1}^d{x}_i{y}_i=x_1{y}_1+x_2{y}_2+\cdots +x_d{y}_d\end{array}$$

where the different operator symbols are just different notations.

Einstein Notation: $s=x_i{y}_i\in F$

Angle Formula

Let $x,y$ be two non-zero vectors, and let $\theta$ be the angle between them, then:

$$⟨x,y⟩=||x||||y||\cos \theta$$

Derivation

We derive formula by comparing two ways to compute $||x-y|{|}^2$.

First, for any vector $v$, its squared length is the dot product of the vector with itself:

$$||v|{|}^2=v_1^2+\cdots +v_d^2=⟨v,v⟩$$

Set $v=x-y$, then:

$$||x-y|{|}^2=⟨x-y,x-y⟩$$

Expand the right-hand side. In coordinates:

$$⟨x-y,x-y⟩=(x_1-y_1{)}^2+\cdots +(x_d-y_d{)}^2$$

Note that each $(x_i-y_i{)}^2$ expands to $(x_i^2-2x_i{y}_i+y_i^2)$. Hence:

$$⟨x-y,x-y⟩=(x_1^2-2x_1{y}_1+y_1^2)+\cdots +(x_d^2-2x_d{y}_d+y_d^2)$$

Given associativity and commutativity, group terms like:

$$⟨x-y,x-y⟩=(x_1^2+\cdots +x_d^2)-2(x_1{y}_1+\cdots +x_d{y}_d)+(y_1^2+\cdots +y_d^2)$$

which can be rewritten as:

$$⟨x-y,x-y⟩=⟨x,x⟩-2⟨x,y⟩+⟨y,y⟩$$

Using the definitions of norm and dot product, this becomes

$$⟨x-y,x-y⟩=⟨x,x⟩-2⟨x,y⟩+⟨y,y⟩.$$ $$⟨x,x-y⟩=⟨x,x⟩-⟨x,y⟩$$

and

$$⟨y,x-y⟩=⟨y,x⟩-⟨y,y⟩.$$

Substituting these back gives

$$⟨x-y,x-y⟩=⟨x,x⟩-⟨x,y⟩-⟨y,x⟩+⟨y,y⟩.$$

For the usual dot product, $⟨x,y⟩=⟨y,x⟩$, so the middle two terms combine:

$$⟨x-y,x-y⟩=⟨x,x⟩-2⟨x,y⟩+⟨y,y⟩.$$

Now use $⟨x,x⟩=\Vert x{\Vert }^2$ and $⟨y,y⟩=\Vert y{\Vert }^2$. This gives

$$\Vert x-y{\Vert }^2=\Vert x{\Vert }^2-2⟨x,y⟩+\Vert y{\Vert }^2.$$

On the other hand, $x$ and $y$ form two sides of a triangle with included angle $\theta$, so the law of cosines gives

$$\Vert x-y{\Vert }^2=\Vert x{\Vert }^2+\Vert y{\Vert }^2-2\Vert x\Vert \Vert y\Vert \cos \theta .$$

These are two expressions for the same quantity, so

$$\Vert x{\Vert }^2-2⟨x,y⟩+\Vert y{\Vert }^2=\Vert x{\Vert }^2+\Vert y{\Vert }^2-2\Vert x\Vert \Vert y\Vert \cos \theta .$$

Cancel $\Vert x{\Vert }^2$ and $\Vert y{\Vert }^2$ from both sides, then divide by $-2$. We get

$$⟨x,y⟩=\Vert x\Vert \Vert y\Vert \cos \theta .$$

Rearranging gives

$$\cos \theta =\frac{⟨x,y⟩}{\Vert x\Vert \Vert y\Vert },$$

so the angle can be recovered from the dot product.

Similarity

The dot product can be viewed as a measurement of similarity of two vectors because it quantifies how much two vectors are aligned. A higher dot product indicates greater degree of alignment or similarity between the vectors.

If $\theta$ is the angle between non-zero vectors $x$ and $y$, then

$$⟨x,y⟩=\Vert x\Vert \Vert y\Vert \cos \theta .$$

So:

• if $\theta =0$, then $\cos \theta =1$ and the dot product is large and positive
• if $\theta =\frac{\pi }{2}$, then $\cos \theta =0$ and the dot product is $0$
• if $\theta =\pi$, then $\cos \theta =-1$ and the dot product is negative

This is why the dot product detects whether two vectors point in similar, orthogonal, or opposite directions.

Projections

Projection onto a Unit Vector

Start with a unit vector $e$. Then $⟨e,x⟩$ is the signed scalar projection of $x$ onto the direction of $e$.

If $\theta$ is the angle between $e$ and $x$, then

$$⟨e,x⟩=\Vert e\Vert \Vert x\Vert \cos \theta .$$

Since $e$ is a unit vector, $\Vert e\Vert =1$, so

$$⟨e,x⟩=\Vert x\Vert \cos \theta .$$

This is exactly the signed length of the part of $x$ that points along $e$.

The corresponding vector projection is

$${\mathrm{proj}}_e(x)=⟨e,x⟩e.$$

For a general non-zero vector $u$, first make it a unit vector:

$$\hat{u}=\frac{u}{\Vert u\Vert }.$$

Then

$$⟨\hat{u},x⟩=\Vert x\Vert \cos \theta ,$$

and the projection onto $u$ is

$${\mathrm{proj}}_u(x)=\frac{⟨u,x⟩}{⟨u,u⟩}u.$$

This is why coefficients in an orthonormal basis can be recovered with the dot product. If

$$x=\sum _i{c}_i{e}_i$$

and the basis vectors satisfy

$$⟨e_i,e_j⟩={\delta }_{ij},$$

then for a fixed $j$,

$$⟨e_j,x⟩=⟨e_j,\sum _i{c}_i{e}_i⟩=\sum _i{c}_i⟨e_j,e_i⟩=\sum _i{c}_i{\delta }_{ji}=c_j.$$

So the dot product with $e_j$ picks out exactly the $j$-th coefficient.