Orthogonality

For $u, v \in \mathbb{R}^n$, the dot product (also called inner product or scalar product) multiplies matching entries and adds the results.

Since $u\cdot u = u_1^2+\cdots+u_n^2$, the Pythagorean theorem tells us this is the square of the length of $u$. That motivates the definition of norm in any dimension.

$u=(4,0)$ stays fixed along the x-axis, with $|u|=4$. Rotate $v$ and change its length to see how the dot product responds, and watch for the angle where it hits zero.

$u$ and $v$ are orthogonal when their dot product is zero. We write $u\perp v$. Being orthogonal is a property of a pair; it does not make sense to call a single vector orthogonal on its own.

Orthogonality is stronger than linear independence: if two nonzero vectors are orthogonal, they are automatically independent, but the converse fails.

A set $\{u_1,\dots,u_r\}$ is orthogonal when every pair of distinct vectors in it is orthogonal. If the set is also a basis of a subspace $U$, we call it an orthogonal basis of $U$; if every vector is additionally normalized, it is an orthonormal basis.

Subspaces $U$ and $V$ are orthogonal to each other, written $U\perp V$, when every vector of one is orthogonal to every vector of the other. It is enough to check this on the basis vectors.

Given a subspace $U\subset\mathbb{R}^n$, the orthogonal complement $U^\perp$ collects every vector orthogonal to all of $U$.

Orthogonal complements connect the four fundamental subspaces of any matrix $A$.

The projection of $u$ over $v$ is the vector along $v$ obtained by "crushing" $u$ onto the line spanned by $v$. This is not symmetric: $\operatorname{proj}_v u$ and $\operatorname{proj}_u v$ are generally different. If $u\perp v$, the projection is 0 either way.

Projecting over $v$ is the same as projecting over any nonzero multiple $\lambda v$, so really we are projecting onto the line $\operatorname{Span}\{v\}$.

Move u (blue) and v (red) with the sliders. The black vector is the projection of u onto v.

When $U$ has an orthogonal basis $\{u_1,\dots,u_r\}$, the coordinates of any $u\in U$ are just its projections onto each basis vector, no linear system needed.

If the basis is orthonormal every denominator is 1, so the coefficients are simply $u\cdot u_i$.

The same idea extends to projecting any vector $v\in\mathbb{R}^n$ (not necessarily in $U$) onto $U$, using an orthogonal basis of $U$.

Every vector splits into its part inside $U$ and its part inside $U^\perp$.

Given a basis $\{u_1,\dots,u_r\}$ of a subspace, not necessarily orthogonal, Gram-Schmidt builds an orthogonal basis $\{\bar u_1,\dots,\bar u_r\}$ of the same subspace. Keep the first vector, then from each later vector subtract its projections onto the vectors already built.

Important: at each step we project onto the vectors already fixed, $\bar u_i$, never onto the original $u_i$. To get an orthonormal basis, add a final step dividing each $\bar u_i$ by its own norm.

A square matrix $A$ is orthogonal when its columns form an orthonormal set: each column has norm 1, and any two different columns are orthogonal to each other.

The name is a bit misleading: what is actually required is orthonormality (unit-length columns included), not just orthogonality between them.

Let $A$ be an orthogonal matrix of size $n\times n$.

A direct consequence is that computing the inverse becomes immediate for these matrices.