Secant Method

Why This Matters

Newton's method requires computing $f'(x)$ at every iteration. In many practical settings, the derivative is expensive, unavailable, or numerically unstable. The secant method replaces the exact derivative with a finite difference approximation using two prior iterates, sacrificing some convergence speed (superlinear instead of quadratic) but eliminating the derivative requirement entirely.

Mental Model

Newton's method uses the tangent line at $x_n$ to find the next iterate. The secant method uses the secant line through $(x_{n-1}, f(x_{n-1}))$ and $(x_n, f(x_n))$. This secant line approximates the tangent line, and the approximation improves as $x_{n-1}$ and $x_n$ converge to the root.

Formal Setup

Definition

Secant Method

Given two initial points $x_0, x_1$ and a function $f$, the secant method generates iterates:

$x_{n+1} = x_n - f(x_n) \cdot \frac{x_n - x_{n-1}}{f(x_n) - f(x_{n-1})}$

The fraction $(f(x_n) - f(x_{n-1}))/(x_n - x_{n-1})$ is the finite difference approximation to $f'(x_n)$.

The method requires two starting points but only one function evaluation per iteration (since $f(x_{n-1})$ was already computed in the previous step). Newton's method requires one starting point but needs both $f(x_n)$ and $f'(x_n)$ per iteration.

Main Theorems

Theorem

Superlinear Convergence of the Secant Method

Statement

Let $r$ be a simple root of $f$ (meaning $f(r) = 0$ and $f'(r) \neq 0$). If $f \in C^2$ near $r$ and the initial points $x_0, x_1$ are sufficiently close to $r$, then the secant method converges with order $\varphi = (1 + \sqrt{5})/2 \approx 1.618$ (the golden ratio):

$|x_{n+1} - r| \leq C |x_n - r|^{\varphi}$

for a constant $C$ depending on $f''(r)/f'(r)$.

Intuition

The convergence order satisfies $p^2 = p + 1$ because each secant step uses information from two previous iterates. Solving this quadratic gives $p = \varphi$. This is strictly between linear ($p = 1$, bisection) and quadratic ($p = 2$, Newton).

Proof Sketch

Define the error $e_n = x_n - r$. Taylor expand $f(x_n) = f'(r)e_n + \frac{1}{2}f''(r)e_n^2 + O(e_n^3)$. Substitute into the secant recurrence and show that $e_{n+1} \approx \frac{f''(r)}{2f'(r)} e_n e_{n-1}$. Setting $|e_{n+1}| \sim C |e_n|^p$ and using $|e_n| \sim C |e_{n-1}|^p$, we get $|e_n|^p \cdot |e_n|^{1/p} \sim |e_n|^{p+1/p}$. Matching exponents gives $p = 1 + 1/p$, so $p^2 = p + 1$, which yields $p = \varphi$.

Why It Matters

The golden ratio convergence order is a clean example of how recycling past information (two points instead of one) can accelerate convergence without additional computation. Per function evaluation, the secant method is often more efficient than Newton because Newton needs both $f$ and $f'$ while secant needs only $f$.

Failure Mode

The method fails if $f(x_n) = f(x_{n-1})$ (division by zero in the secant formula). It also has no guaranteed convergence from arbitrary starting points, unlike bisection. If the starting points are far from the root or $f'(r) = 0$ (multiple root), convergence degrades or fails entirely.

Comparison With Other Methods

Method	Convergence order	Derivatives needed	Evaluations per step	Guaranteed convergence
Bisection	1 (linear)	None	1	Yes (if bracket exists)
Secant	~1.618 (superlinear)	None	1	No
Newton	2 (quadratic)	$f'$	2 ($f$ and $f'$)	No

Efficiency per evaluation: Newton converges in fewer iterations, but each iteration costs two function evaluations (counting $f'$ as one evaluation). The secant method uses one evaluation per step. The efficiency index (convergence order per evaluation) is $2^{1/2} \approx 1.41$ for Newton and $\varphi^{1/1} \approx 1.62$ for secant. By this measure, secant is more efficient.

Canonical Examples

Example

Solving x^3 - 2 = 0

Starting from $x_0 = 1, x_1 = 2$ with $f(x) = x^3 - 2$:

• $x_2 = 2 - 6 \cdot \frac{2 - 1}{6 - (-1)} = 2 - 6/7 \approx 1.1429$
• $x_3 \approx 1.2536$
• $x_4 \approx 1.2599$ (true root: $\sqrt[3]{2} \approx 1.25992$)

Convergence to 4 decimal places in 4 iterations, with no derivative computation.

Common Confusions

Watch Out

Secant method is not the same as the method of false position

The method of false position (regula falsi) also uses a secant line, but it maintains a bracket $[a, b]$ with $f(a)f(b) < 0$, guaranteeing convergence. The secant method always uses the two most recent iterates, which can both be on the same side of the root. Regula falsi sacrifices convergence speed for safety.

Watch Out

Superlinear does not mean faster than Newton in iterations

The secant method takes more iterations than Newton to reach the same accuracy. The advantage is per-function-evaluation efficiency, not per-iteration speed. If $f'$ is cheap (e.g., automatic differentiation is available), Newton is usually preferable.

Exercises

ExerciseCore

Problem

Derive the secant update formula by replacing $f'(x_n)$ in Newton's method $x_{n+1} = x_n - f(x_n)/f'(x_n)$ with the finite difference approximation using $x_{n-1}$ and $x_n$.

Hint

Reveal Solution

ExerciseAdvanced

Problem

Show that the convergence order $p$ of the secant method satisfies $p^2 - p - 1 = 0$, and explain why this equation arises from the error recurrence $e_{n+1} \approx C \cdot e_n \cdot e_{n-1}$.

Hint

Reveal Solution

References

Canonical:

• Burden & Faires, Numerical Analysis, 10th ed., Chapter 2.3
• Stoer & Bulirsch, Introduction to Numerical Analysis, Chapter 5

Current:

• Nocedal & Wright, Numerical Optimization (2006), Chapter 11

• Hastie, Tibshirani, Friedman, The Elements of Statistical Learning (2009)

Next Topics

• Quasi-Newton methods: generalize the secant idea to multidimensional optimization (BFGS, L-BFGS)