Taylor Approximation

Near a point, a smooth function is approximated by a polynomial built from its derivatives there; first order is the tangent, second adds curvature.

5 min read564 wordscore6 viewsupdated 2026-10-03

Definition

The Taylor polynomial of order nn of a function ff at a point aa is the unique polynomial TnT_n of degree at most nn whose value and first nn derivatives at aa equal those of ff: Tn(x)=∑k=0nf(k)(a)k!(x−a)kT_n(x)=\sum_{k=0}^{n}\frac{f^{(k)}(a)}{k!}(x-a)^k. Near aa it approximates ff, with an error that shrinks like ∣x−a∣n+1|x-a|^{n+1}.

At a glance

Notation
Tn(x)T_n(x) or Pn(x)P_n(x); remainder Rn(x)=f(x)−Tn(x)R_n(x)=f(x)-T_n(x); f(k)f^{(k)} is the kk-th derivative
Input → output
a smooth function, a point aa and an order nn → a degree-nn polynomial
Key property
error =O(∣x−a∣n+1)=O(|x-a|^{n+1}) near aa; first order is the tangent line, second order adds curvature
Multivariate
f(θ+δ)≈f(θ)+∇f⊤δ+12δ⊤Hδf(\theta+\delta)\approx f(\theta)+\nabla f^\top\delta+\tfrac12\delta^\top H\delta
Prerequisites
derivatives & gradients, factorials
Used in
gradient descent and Newton's method, XGBoost, the delta method, numerically stable functions

The intuition

Zoom in on a smooth curve and it looks like a straight line — the tangent, f(a)+f′(a)(x−a)f(a)+f'(a)(x-a). Zoom out a little and the line starts to miss because the curve bends. Adding a quadratic term with the right curvature, 12f′′(a)(x−a)2\tfrac12f''(a)(x-a)^2, fixes the bend; a cubic term fixes how the bend changes, and so on.

Each term matches one more derivative at aa, so the polynomial agrees with ff ever more closely near aa. Far from aa, a polynomial can only do what polynomials do — eventually it shoots off to ±∞\pm\infty — so a Taylor approximation is a local statement.

The math

Tn(x)=∑k=0nf(k)(a)k!(x−a)k=f(a)+f′(a)(x−a)+f′′(a)2(x−a)2+…T_n(x)=\sum_{k=0}^{n}\frac{f^{(k)}(a)}{k!}(x-a)^k=f(a)+f'(a)(x-a)+\frac{f''(a)}{2}(x-a)^2+\dots
aa
expansion point, where the derivatives are evaluated
f(k)(a)f^{(k)}(a)
kk-th derivative of ff at aa (f(0)=ff^{(0)}=f)
k!k!
1⋅2⋯k1\cdot2\cdots k, with 0!=10!=1

Why the factorials. Differentiating (x−a)k(x-a)^k exactly kk times gives k!k!, and every other power vanishes at x=ax=a after kk derivatives. So the kk-th derivative of TnT_n at aa is ck k!c_k\,k!, which matches f(k)(a)f^{(k)}(a) exactly when ck=f(k)(a)/k!c_k=f^{(k)}(a)/k!.

How big the error is. If ff has n+1n+1 continuous derivatives, the Lagrange form of the remainder says that for some ξ\xi between aa and xx:

Rn(x)=f(x)−Tn(x)=f(n+1)(ξ)(n+1)!(x−a)n+1,∣Rn(x)∣≤M(n+1)!∣x−a∣n+1R_n(x)=f(x)-T_n(x)=\frac{f^{(n+1)}(\xi)}{(n+1)!}(x-a)^{n+1},\qquad |R_n(x)|\le\frac{M}{(n+1)!}|x-a|^{n+1}
ξ\xi
some unknown point between aa and xx
MM
a bound on ∣f(n+1)∣|f^{(n+1)}| over that interval

Halving ∣x−a∣|x-a| divides the error bound by 2n+12^{n+1}. Standard expansions at a=0a=0: ex=1+x+x22+x36+…e^x=1+x+\frac{x^2}{2}+\frac{x^3}{6}+\dots, sin⁡x=x−x36+x5120−…\sin x=x-\frac{x^3}{6}+\frac{x^5}{120}-\dots, ln⁡(1+x)=x−x22+x33−…\ln(1+x)=x-\frac{x^2}{2}+\frac{x^3}{3}-\dots

Convergence is not guaranteed. Letting n→∞n\to\infty gives the Taylor series. For exe^x and sin⁡x\sin x it converges everywhere. For ln⁡(1+x)\ln(1+x) it converges only for −1<x≤1-1<x\le1 when a=0a=0 — in general only within distance 1+a1+a of aa, the distance to the singularity at x=−1x=-1. Outside that radius, more terms make the approximation worse.

Several variables. For f:Rn→Rf:\mathbb{R}^n\to\mathbb{R}, the second-order expansion uses the gradient ∇f\nabla f and the Hessian HH (matrix of second partials). Minimizing this quadratic model over δ\delta gives Newton's step; minimizing only the linear part plus a penalty 12η∥δ∥2\frac{1}{2\eta}\lVert\delta\rVert^2 gives the gradient-descent step:

f(θ+δ)≈f(θ)+∇f(θ)⊤δ+12 δ⊤H(θ) δ ⇒ δNewton=−H−1∇f,δGD=−η ∇ff(\theta+\delta)\approx f(\theta)+\nabla f(\theta)^\top\delta+\tfrac12\,\delta^\top H(\theta)\,\delta\ \Rightarrow\ \delta_{\text{Newton}}=-H^{-1}\nabla f,\quad \delta_{\text{GD}}=-\eta\,\nabla f
δ\delta
step in parameter space
HH
Hessian, Hij=∂2f/∂θi∂θjH_{ij}=\partial^2f/\partial\theta_i\partial\theta_j

See it

drag a and x on the plot
xa
T3(x)=x−0.167x3T_{3}(x) = x - 0.167x^{3}
a = 0.00x = 1.60f(x) = 0.9996Tₙ(x) = 0.9173error f − Tₙ = 0.0822▬ f · ▬ Tₙ · ▬ |error| < 0.01
fig.Start: sin x around a = 0 with n = 3, so T₃ = x − x³/6. The green band marks |error| < 0.01: |x| < 1.04 at n = 3, widening to 1.76 at n = 5 and 2.50 at n = 7. At the probe x = 1.6 the cubic gives 0.9173 against sin 1.6 = 0.9996. For ln(1+x), past the dashed line x = 1 + 2a raising the order makes things worse: at a = 0 and x = 1.5, T₈ is off by 1.82.

Worked example

Approximate e0.5=1.648721e^{0.5}=1.648721 around a=0a=0, where every derivative of exe^x equals 1. T1=1+0.5=1.5T_1=1+0.5=1.5 (error 0.14870.1487); T2=1.5+0.125=1.625T_2=1.5+0.125=1.625 (error 0.02370.0237); T3=1.625+0.020833=1.645833T_3=1.625+0.020833=1.645833 (error 0.002890.00289). Each extra order cuts the error by roughly a factor of 6–8.

The Lagrange bound for n=3n=3 uses M=e0.5M=e^{0.5} (the largest fourth derivative on [0,0.5][0,0.5]): ∣R3∣≤1.6487⋅0.54/24=0.00429|R_3|\le1.6487\cdot0.5^4/24=0.00429, which indeed exceeds the true 0.002890.00289.

Newton's method applies the second-order model repeatedly. To minimize f(t)=et−2tf(t)=e^t-2t (minimum at ln⁡2=0.693147\ln2=0.693147), each step is t←t−f′(t)/f′′(t)=t−(et−2)/ett\leftarrow t-f'(t)/f''(t)=t-(e^t-2)/e^t. From t=0t=0: 11, 0.7357590.735759, 0.6940420.694042, 0.6931480.693148 — the number of correct digits roughly doubles each step.

In code

python
1import numpy as np
2from math import factorial
3
4def taylor_exp(x, n, a=0.0): # every derivative of e^x is e^x
5 return sum(np.exp(a) * (x - a)**k / factorial(k) for k in range(n + 1))
6
7for n in (1, 2, 3):
8 print(n, taylor_exp(0.5, n), np.exp(0.5) - taylor_exp(0.5, n))
9# 1 1.5 0.1487
10# 2 1.625 0.0237
11# 3 1.6458 0.0029
12
13t = 0.0 # Newton on f(t) = e^t - 2t
14for _ in range(4):
15 t -= (np.exp(t) - 2) / np.exp(t) # f'(t) / f''(t)
16print(t, np.log(2)) # 0.693147... 0.693147...
17
18print(np.log(1 + 1e-17), np.log1p(1e-17)) # 0.0 1e-17 (log1p uses ln(1+x) ≈ x - x²/2 + …)
Taylor polynomials of eˣ, Newton's method from the quadratic model, and why log1p exists.

Where it's used

Optimizer design

Gradient descent trusts the first-order model inside a small step; Newton and quasi-Newton methods (L-BFGS) minimize the second-order model; the step-size limit η<2/λmax⁡\eta<2/\lambda_{\max} comes from the quadratic term.

Gradient boosting

XGBoost and LightGBM expand the loss to second order around the current prediction, so each tree is fit using only per-sample gradients gig_i and Hessians hih_i, with leaf value −∑gi/(∑hi+λ)-\sum g_i/(\sum h_i+\lambda).

The delta method

A first-order expansion g(θ^)≈g(θ)+g′(θ)(θ^−θ)g(\hat\theta)\approx g(\theta)+g'(\theta)(\hat\theta-\theta) gives the standard error of a transformed estimate: SE(g(θ^))≈∣g′(θ)∣ SE(θ^)\mathrm{SE}(g(\hat\theta))\approx|g'(\theta)|\,\mathrm{SE}(\hat\theta).

Fast, stable numerics

log1p and expm1 exist because ln⁡(1+x)≈x\ln(1+x)\approx x and ex−1≈xe^x-1\approx x for tiny xx, where the naive formula loses every digit; second-order expansion also underlies the analysis of how pruning or quantizing weights changes the loss.

Check yourself

What is the second-order Taylor approximation of 1+x\sqrt{1+x} at a=0a=0, evaluated at x=0.1x=0.1?

Check yourself

For f(x)=x2f(x)=x^2, the second-order model is exact. How many Newton steps does it take to reach the minimum from any start?

Remember

  • ✓Tn(x)=∑k≤nf(k)(a)k!(x−a)kT_n(x)=\sum_{k\le n}\frac{f^{(k)}(a)}{k!}(x-a)^k matches ff's first nn derivatives at aa.
  • ✓Error is O(∣x−a∣n+1)O(|x-a|^{n+1}) near aa; the Lagrange remainder bounds it.
  • ✓First order gives gradient descent; second order (with the Hessian) gives Newton's method.
  • ✓It is local: outside the radius of convergence, more terms make it worse.
← back to Mathematics roadmap