Continuity in R^n

Why This Matters

Continuity is the minimal regularity condition on functions. Without it, optimization is hopeless: you cannot guarantee that gradient descent approaches a minimum, that a loss function attains its infimum on a compact set, or that small changes in parameters produce small changes in predictions.

Lipschitz continuity is the variant that appears most in ML theory. Generalization bounds for neural networks often depend on the Lipschitz constant of the network. Wasserstein GANs enforce a Lipschitz constraint explicitly.

Core Definitions

Definition

Continuity at a Point

A function $f: \mathbb{R}^n \to \mathbb{R}^m$ (where $\mathbb{R}^n$ carries the standard metric) is continuous at $a \in \mathbb{R}^n$ if for every $\epsilon > 0$, there exists $\delta > 0$ such that:

$\|x - a\| < \delta \implies \|f(x) - f(a)\| < \epsilon$

The choice of $\delta$ can depend on both $\epsilon$ and the point $a$.

Definition

Uniform Continuity

$f: \mathbb{R}^n \to \mathbb{R}^m$ is uniformly continuous on a set $S$ if for every $\epsilon > 0$, there exists $\delta > 0$ such that for all $x, y \in S$:

$\|x - y\| < \delta \implies \|f(x) - f(y)\| < \epsilon$

The $\delta$ depends only on $\epsilon$, not on the specific points $x, y$.

Definition

Lipschitz Continuity

$f: \mathbb{R}^n \to \mathbb{R}^m$ is $L$-Lipschitz on a set $S$ if there exists $L \geq 0$ such that for all $x, y \in S$:

$\|f(x) - f(y)\| \leq L \|x - y\|$

The smallest such $L$ is the Lipschitz constant of $f$ on $S$.

The hierarchy is strict: Lipschitz implies uniformly continuous implies continuous. The converses fail. The function $f(x) = \sqrt{x}$ on $[0,1]$ is uniformly continuous but not Lipschitz (its derivative blows up at 0). The function $f(x) = x^2$ on $\mathbb{R}$ is continuous but not uniformly continuous.

Composition and Algebraic Properties

Continuous functions compose: if $f$ is continuous at $a$ and $g$ is continuous at $f(a)$, then $g \circ f$ is continuous at $a$. Sums, products, and quotients (where the denominator is nonzero) of continuous functions are continuous.

For Lipschitz functions, the composition rule is quantitative. If $f$ is $L_f$-Lipschitz and $g$ is $L_g$-Lipschitz, then $g \circ f$ is $L_f L_g$-Lipschitz. This multiplicative blowup is why deep networks can have large Lipschitz constants: each layer multiplies.

Main Theorems

Theorem

Extreme Value Theorem

Statement

If $f: S \to \mathbb{R}$ is continuous and $S \subseteq \mathbb{R}^n$ is compact and nonempty, then $f$ attains its maximum and minimum on $S$. That is, there exist $x_{\min}, x_{\max} \in S$ such that:

$f(x_{\min}) \leq f(x) \leq f(x_{\max}) \quad \text{for all } x \in S$

Intuition

A continuous function on a closed and bounded set cannot "escape to infinity" or "approach but never reach" its supremum. Compactness traps sequences and continuity preserves limits.

Proof Sketch

Since $f$ is continuous and $S$ is compact, $f(S)$ is compact in $\mathbb{R}$ (continuous image of a compact set is compact). A compact subset of $\mathbb{R}$ is closed and bounded, so it contains its supremum and infimum.

Why It Matters

This theorem guarantees that optimization problems over compact sets have solutions. When you minimize a continuous loss over a bounded parameter space, a minimizer exists. Without compactness, minimizers may not exist: $\inf_{x > 0} 1/x = 0$ but no $x > 0$ achieves it.

Failure Mode

Fails without compactness. On the open interval $(0,1)$, the function $f(x) = 1/x$ is continuous but has no maximum. Fails without continuity: the indicator function $\mathbf{1}_{\{0\}}$ on $[-1,1]$ achieves its max but a discontinuous function in general need not.

Theorem

Heine-Cantor Theorem

Statement

If $f: S \to \mathbb{R}^m$ is continuous and $S \subseteq \mathbb{R}^n$ is compact, then $f$ is uniformly continuous on $S$.

Intuition

On a compact set, continuity cannot degrade from point to point. The worst-case $\delta$ over all points in $S$ is still positive because $S$ has no "escape to infinity" where the modulus of continuity might shrink to zero.

Proof Sketch

Suppose $f$ is not uniformly continuous. Then there exists $\epsilon > 0$ and sequences $x_k, y_k$ with $\|x_k - y_k\| \to 0$ but $\|f(x_k) - f(y_k)\| \geq \epsilon$. By compactness, extract a convergent subsequence $x_{k_j} \to a$. Then $y_{k_j} \to a$ as well. By continuity at $a$, $\|f(x_{k_j}) - f(y_{k_j})\| \to 0$, contradicting $\geq \epsilon$.

Why It Matters

This is why bounded parameter spaces simplify analysis. A continuous loss function on a compact parameter set is automatically uniformly continuous, which makes approximation arguments (like discretizing the parameter space) valid.

Failure Mode

Fails on non-compact domains. $f(x) = x^2$ is continuous on $\mathbb{R}$ but not uniformly continuous: for large $x$, a small change in $x$ produces a large change in $x^2$.

Common Confusions

Watch Out

Lipschitz constant depends on the norm

The Lipschitz constant of a function depends on which norm you use. A function that is 1-Lipschitz in the $\ell_2$ norm may have a different Lipschitz constant in the $\ell_\infty$ norm. In ML, the $\ell_2$ norm is the default unless stated otherwise.

Watch Out

Differentiable does not imply Lipschitz

A function can be differentiable everywhere without being Lipschitz. $f(x) = x^2$ on $\mathbb{R}$ is smooth but not Lipschitz because its derivative is unbounded. Lipschitz on $\mathbb{R}^n$ requires that $\|\nabla f\|$ be bounded.

Exercises

ExerciseCore

Problem

Prove that $f(x) = \|x\|_2$ is 1-Lipschitz on $\mathbb{R}^n$.

Hint

Reveal Solution

ExerciseAdvanced

Problem

Let $f: \mathbb{R}^n \to \mathbb{R}$ be differentiable with $\|\nabla f(x)\| \leq L$ for all $x$. Prove that $f$ is $L$-Lipschitz.

Hint

Reveal Solution

References

Canonical:

• Rudin, Principles of Mathematical Analysis (1976), Chapters 4 and 7
• Apostol, Mathematical Analysis (1974), Chapter 4
• Folland, Real Analysis (1999), Chapter 4 (continuity and topology)

Current:

• Shalev-Shwartz & Ben-David, Understanding Machine Learning (2014), Section 26.1 (Lipschitz conditions in generalization)
• Vershynin, High-Dimensional Probability (2018), Section 5.2.2 (Lipschitz functions and concentration)
• Deisenroth, Faisal, Ong, Mathematics for Machine Learning (2020), Section 5.1 (continuity in the context of differentiation)