# Taylor Approximation

> Near a point, a smooth function is approximated by a polynomial built from its derivatives there; first order is the tangent, second adds curvature.

Source: https://www.mlfoundry.in/learn/taylor-approximation  
Section: Mathematics › Calculus & Optimization  
Level: core · 5 min read · updated 2026-10-03

**The Taylor polynomial:** of order $n$ of a function $f$ at a point $a$ is the unique polynomial $T_n$ of degree at most $n$ whose value and first $n$ derivatives at $a$ equal those of $f$: $T_n(x)=\sum_{k=0}^{n}\frac{f^{(k)}(a)}{k!}(x-a)^k$. Near $a$ it approximates $f$, with an error that shrinks like $|x-a|^{n+1}$.

- **Notation:** $T_n(x)$ or $P_n(x)$; remainder $R_n(x)=f(x)-T_n(x)$; $f^{(k)}$ is the $k$-th derivative
- **Input → output:** a smooth function, a point $a$ and an order $n$ → a degree-$n$ polynomial
- **Key property:** error $=O(|x-a|^{n+1})$ near $a$; first order is the tangent line, second order adds curvature
- **Multivariate:** $f(\theta+\delta)\approx f(\theta)+\nabla f^\top\delta+\tfrac12\delta^\top H\delta$
- **Prerequisites:** derivatives & gradients, factorials
- **Used in:** gradient descent and Newton's method, XGBoost, the delta method, numerically stable functions

## The intuition

Zoom in on a smooth curve and it looks like a straight line — the tangent, $f(a)+f'(a)(x-a)$. Zoom out a little and the line starts to miss because the curve bends. Adding a quadratic term with the right curvature, $\tfrac12f''(a)(x-a)^2$, fixes the bend; a cubic term fixes how the bend changes, and so on.

Each term matches one more derivative at $a$, so the polynomial agrees with $f$ ever more closely near $a$. Far from $a$, a polynomial can only do what polynomials do — eventually it shoots off to $\pm\infty$ — so a Taylor approximation is a **local** statement.

## The math

$$
T_n(x)=\sum_{k=0}^{n}\frac{f^{(k)}(a)}{k!}(x-a)^k=f(a)+f'(a)(x-a)+\frac{f''(a)}{2}(x-a)^2+\dots
$$

- $a$: expansion point, where the derivatives are evaluated
- $f^{(k)}(a)$: $k$-th derivative of $f$ at $a$ ($f^{(0)}=f$)
- $k!$: $1\cdot2\cdots k$, with $0!=1$

**Why the factorials.** Differentiating $(x-a)^k$ exactly $k$ times gives $k!$, and every other power vanishes at $x=a$ after $k$ derivatives. So the $k$-th derivative of $T_n$ at $a$ is $c_k\,k!$, which matches $f^{(k)}(a)$ exactly when $c_k=f^{(k)}(a)/k!$.

**How big the error is.** If $f$ has $n+1$ continuous derivatives, the Lagrange form of the remainder says that for some $\xi$ between $a$ and $x$:

$$
R_n(x)=f(x)-T_n(x)=\frac{f^{(n+1)}(\xi)}{(n+1)!}(x-a)^{n+1},\qquad |R_n(x)|\le\frac{M}{(n+1)!}|x-a|^{n+1}
$$

- $\xi$: some unknown point between $a$ and $x$
- $M$: a bound on $|f^{(n+1)}|$ over that interval

Halving $|x-a|$ divides the error bound by $2^{n+1}$. Standard expansions at $a=0$: $e^x=1+x+\frac{x^2}{2}+\frac{x^3}{6}+\dots$, $\sin x=x-\frac{x^3}{6}+\frac{x^5}{120}-\dots$, $\ln(1+x)=x-\frac{x^2}{2}+\frac{x^3}{3}-\dots$

**Convergence is not guaranteed.** Letting $n\to\infty$ gives the Taylor series. For $e^x$ and $\sin x$ it converges everywhere. For $\ln(1+x)$ it converges only for $-1<x\le1$ when $a=0$ — in general only within distance $1+a$ of $a$, the distance to the singularity at $x=-1$. Outside that radius, more terms make the approximation worse.

**Several variables.** For $f:\mathbb{R}^n\to\mathbb{R}$, the second-order expansion uses the gradient $\nabla f$ and the Hessian $H$ (matrix of second partials). Minimizing this quadratic model over $\delta$ gives **Newton's step**; minimizing only the linear part plus a penalty $\frac{1}{2\eta}\lVert\delta\rVert^2$ gives the **gradient-descent step**:

$$
f(\theta+\delta)\approx f(\theta)+\nabla f(\theta)^\top\delta+\tfrac12\,\delta^\top H(\theta)\,\delta\ \Rightarrow\ \delta_{\text{Newton}}=-H^{-1}\nabla f,\quad \delta_{\text{GD}}=-\eta\,\nabla f
$$

- $\delta$: step in parameter space
- $H$: Hessian, $H_{ij}=\partial^2f/\partial\theta_i\partial\theta_j$

## See it

> Figure: Start: sin x around a = 0 with n = 3, so T₃ = x − x³/6. The green band marks |error| < 0.01: |x| < 1.04 at n = 3, widening to 1.76 at n = 5 and 2.50 at n = 7. At the probe x = 1.6 the cubic gives 0.9173 against sin 1.6 = 0.9996. For ln(1+x), past the dashed line x = 1 + 2a raising the order makes things worse: at a = 0 and x = 1.5, T₈ is off by 1.82.

## Worked example

Approximate $e^{0.5}=1.648721$ around $a=0$, where every derivative of $e^x$ equals 1. $T_1=1+0.5=1.5$ (error $0.1487$); $T_2=1.5+0.125=1.625$ (error $0.0237$); $T_3=1.625+0.020833=1.645833$ (error $0.00289$). Each extra order cuts the error by roughly a factor of 6–8.

The Lagrange bound for $n=3$ uses $M=e^{0.5}$ (the largest fourth derivative on $[0,0.5]$): $|R_3|\le1.6487\cdot0.5^4/24=0.00429$, which indeed exceeds the true $0.00289$.

Newton's method applies the second-order model repeatedly. To minimize $f(t)=e^t-2t$ (minimum at $\ln2=0.693147$), each step is $t\leftarrow t-f'(t)/f''(t)=t-(e^t-2)/e^t$. From $t=0$: $1$, $0.735759$, $0.694042$, $0.693148$ — the number of correct digits roughly doubles each step.

## In code

```python
import numpy as np
from math import factorial

def taylor_exp(x, n, a=0.0):   # every derivative of e^x is e^x
    return sum(np.exp(a) * (x - a)**k / factorial(k) for k in range(n + 1))

for n in (1, 2, 3):
    print(n, taylor_exp(0.5, n), np.exp(0.5) - taylor_exp(0.5, n))
# 1 1.5      0.1487
# 2 1.625    0.0237
# 3 1.6458   0.0029

t = 0.0                         # Newton on f(t) = e^t - 2t
for _ in range(4):
    t -= (np.exp(t) - 2) / np.exp(t)   # f'(t) / f''(t)
print(t, np.log(2))             # 0.693147... 0.693147...

print(np.log(1 + 1e-17), np.log1p(1e-17))   # 0.0 1e-17  (log1p uses ln(1+x) ≈ x - x²/2 + …)
```

Taylor polynomials of eˣ, Newton's method from the quadratic model, and why log1p exists.

- **Optimizer design:** Gradient descent trusts the first-order model inside a small step; Newton and quasi-Newton methods (L-BFGS) minimize the second-order model; the step-size limit $\eta<2/\lambda_{\max}$ comes from the quadratic term.
- **Gradient boosting:** XGBoost and LightGBM expand the loss to second order around the current prediction, so each tree is fit using only per-sample gradients $g_i$ and Hessians $h_i$, with leaf value $-\sum g_i/(\sum h_i+\lambda)$.
- **The delta method:** A first-order expansion $g(\hat\theta)\approx g(\theta)+g'(\theta)(\hat\theta-\theta)$ gives the standard error of a transformed estimate: $\mathrm{SE}(g(\hat\theta))\approx|g'(\theta)|\,\mathrm{SE}(\hat\theta)$.
- **Fast, stable numerics:** log1p and expm1 exist because $\ln(1+x)\approx x$ and $e^x-1\approx x$ for tiny $x$, where the naive formula loses every digit; second-order expansion also underlies the analysis of how pruning or quantizing weights changes the loss.

> **Higher order is not always better:** Adding terms improves the approximation only inside the radius of convergence. For $\ln(1+x)$ at $a=0$ and $x=1.5$, the errors for $n=2,4,8$ are $0.54$, $0.68$ and $1.82$ — growing. Even inside the radius, a high-order polynomial is only trustworthy near $a$; this is why Newton steps are damped or limited to a trust region.

> **Smooth is required:** The expansion needs the derivatives to exist. ReLU, $|x|$ and the hinge loss have a kink, so no quadratic model describes them across it. And a function can have all derivatives yet a useless series: $e^{-1/x^2}$ (with value 0 at 0) has every derivative 0 at $x=0$, so its Taylor series is identically zero.

**Question:** What is the second-order Taylor approximation of $\sqrt{1+x}$ at $a=0$, evaluated at $x=0.1$?

- A. 1.05
- B. 1.04875
- C. 1.05125
- D. 1.0488088

**Answer:** B. 1.04875 — $f(0)=1$, $f'(0)=\tfrac12$, $f''(0)=-\tfrac14$, so $T_2=1+\tfrac{x}{2}-\tfrac{x^2}{8}=1+0.05-0.00125=1.04875$. The true value $\sqrt{1.1}=1.0488088$ differs by $6\times10^{-5}$, roughly the size of the next term $x^3/16$.

**Question:** For $f(x)=x^2$, the second-order model is exact. How many Newton steps does it take to reach the minimum from any start?

- A. 1
- B. 2
- C. about log of the distance
- D. it never reaches it exactly

**Answer:** A. 1 — Newton's step is $x-f'(x)/f''(x)=x-2x/2=0$. Because $T_2$ equals $f$ exactly for a quadratic, minimizing the model minimizes $f$ in one step.

**Key points**

- $T_n(x)=\sum_{k\le n}\frac{f^{(k)}(a)}{k!}(x-a)^k$ matches $f$'s first $n$ derivatives at $a$.
- Error is $O(|x-a|^{n+1})$ near $a$; the Lagrange remainder bounds it.
- First order gives gradient descent; second order (with the Hessian) gives Newton's method.
- It is local: outside the radius of convergence, more terms make it worse.

---

From MLFoundry (https://www.mlfoundry.in). The web page adds interactive figures and diagrams.
