Definition
The Taylor polynomial of order of a function at a point is the unique polynomial of degree at most whose value and first derivatives at equal those of : . Near it approximates , with an error that shrinks like .
At a glance
- Notation
- or ; remainder ; is the -th derivative
- Input → output
- a smooth function, a point and an order → a degree- polynomial
- Key property
- error near ; first order is the tangent line, second order adds curvature
- Multivariate
- Prerequisites
- derivatives & gradients, factorials
- Used in
- gradient descent and Newton's method, XGBoost, the delta method, numerically stable functions
The intuition
Zoom in on a smooth curve and it looks like a straight line — the tangent, . Zoom out a little and the line starts to miss because the curve bends. Adding a quadratic term with the right curvature, , fixes the bend; a cubic term fixes how the bend changes, and so on.
Each term matches one more derivative at , so the polynomial agrees with ever more closely near . Far from , a polynomial can only do what polynomials do — eventually it shoots off to — so a Taylor approximation is a local statement.
The math
- expansion point, where the derivatives are evaluated
- -th derivative of at ()
- , with
Why the factorials. Differentiating exactly times gives , and every other power vanishes at after derivatives. So the -th derivative of at is , which matches exactly when .
How big the error is. If has continuous derivatives, the Lagrange form of the remainder says that for some between and :
- some unknown point between and
- a bound on over that interval
Halving divides the error bound by . Standard expansions at : , ,
Convergence is not guaranteed. Letting gives the Taylor series. For and it converges everywhere. For it converges only for when — in general only within distance of , the distance to the singularity at . Outside that radius, more terms make the approximation worse.
Several variables. For , the second-order expansion uses the gradient and the Hessian (matrix of second partials). Minimizing this quadratic model over gives Newton's step; minimizing only the linear part plus a penalty gives the gradient-descent step:
- step in parameter space
- Hessian,
See it
Worked example
Approximate around , where every derivative of equals 1. (error ); (error ); (error ). Each extra order cuts the error by roughly a factor of 6–8.
The Lagrange bound for uses (the largest fourth derivative on ): , which indeed exceeds the true .
Newton's method applies the second-order model repeatedly. To minimize (minimum at ), each step is . From : , , , — the number of correct digits roughly doubles each step.
In code
1import numpy as np2from math import factorial34def taylor_exp(x, n, a=0.0): # every derivative of e^x is e^x5 return sum(np.exp(a) * (x - a)**k / factorial(k) for k in range(n + 1))67for n in (1, 2, 3):8 print(n, taylor_exp(0.5, n), np.exp(0.5) - taylor_exp(0.5, n))9# 1 1.5 0.148710# 2 1.625 0.023711# 3 1.6458 0.00291213t = 0.0 # Newton on f(t) = e^t - 2t14for _ in range(4):15 t -= (np.exp(t) - 2) / np.exp(t) # f'(t) / f''(t)16print(t, np.log(2)) # 0.693147... 0.693147...1718print(np.log(1 + 1e-17), np.log1p(1e-17)) # 0.0 1e-17 (log1p uses ln(1+x) ≈ x - x²/2 + …)Where it's used
Optimizer design
Gradient descent trusts the first-order model inside a small step; Newton and quasi-Newton methods (L-BFGS) minimize the second-order model; the step-size limit comes from the quadratic term.
Gradient boosting
XGBoost and LightGBM expand the loss to second order around the current prediction, so each tree is fit using only per-sample gradients and Hessians , with leaf value .
The delta method
A first-order expansion gives the standard error of a transformed estimate: .
Fast, stable numerics
log1p and expm1 exist because and for tiny , where the naive formula loses every digit; second-order expansion also underlies the analysis of how pruning or quantizing weights changes the loss.
Check yourself
What is the second-order Taylor approximation of at , evaluated at ?
Check yourself
For , the second-order model is exact. How many Newton steps does it take to reach the minimum from any start?
Remember
- ✓ matches 's first derivatives at .
- ✓Error is near ; the Lagrange remainder bounds it.
- ✓First order gives gradient descent; second order (with the Hessian) gives Newton's method.
- ✓It is local: outside the radius of convergence, more terms make it worse.