Inner Product Spaces and Orthogonality

An inner product on a real vector space $V$ is a function $\langle \cdot, \cdot \rangle: V \times V \to \mathbb{R}$ satisfying:

1. Symmetry: $\langle u, v \rangle = \langle v, u \rangle$
2. Linearity in first argument: $\langle \alpha u + \beta v, w \rangle = \alpha \langle u, w \rangle + \beta \langle v, w \rangle$
3. Positive definiteness: $\langle v, v \rangle \geq 0$ with equality iff $v = 0$

The induced norm is $\|v\| = \sqrt{\langle v, v \rangle}$. The standard inner product on $\mathbb{R}^n$ is $\langle x, y \rangle = x^T y$.

Two vectors $u, v$ are orthogonal if $\langle u, v \rangle = 0$, written $u \perp v$. A set $\{e_1, \ldots, e_k\}$ is orthonormal if $\langle e_i, e_j \rangle = \delta_{ij}$ (Kronecker delta). The orthogonal complement of a subspace $W$ is $W^\perp = \{v \in V : \langle v, w \rangle = 0 \text{ for all } w \in W\}$.

The orthogonal projection of $v$ onto a subspace $W$ is the unique $\hat{v} \in W$ minimizing $\|v - w\|$ over $w \in W$. If $\{e_1, \ldots, e_k\}$ is an orthonormal basis for $W$:

$\text{proj}_W(v) = \sum_{i=1}^{k} \langle v, e_i \rangle \, e_i$

The residual $v - \text{proj}_W(v)$ lies in $W^\perp$.

A Hilbert space is a complete inner product space: every Cauchy sequence converges. Finite-dimensional inner product spaces are automatically Hilbert spaces. Infinite-dimensional examples include $L^2$ (the space of square-integrable functions) and reproducing kernel Hilbert spaces (RKHS) used in kernel methods.