Neural ODE (Neural Ordinary Differential Equation)

A ResNet is Euler's method; take the step to zero and the network becomes a learned vector field whose forward pass is an ODE solve, trained with constant memory by the adjoint method.

11 min read936 wordsadvanced0 viewsupdated 2026-10-11

Definition

Neural ODE is a model whose hidden state evolves continuously according to dh/dt=fθ(h(t),t)dh/dt=f_\theta(h(t),t), where fθf_\theta is a neural network. The output is h(t1)h(t_1), obtained by running a numerical ODE solver from h(t0)=xh(t_0)=x; gradients come from solving a second ODE backward in time (the adjoint method).

The idea was popularised by Neural Ordinary Differential Equations (Ricky T. Q. Chen, Yulia Rubanova, Jesse Bettencourt and David Duvenaud), a best-paper award winner at NeurIPS 2018. The view of residual networks as discretised dynamical systems was already around (E 2017; Haber & Ruthotto 2017; Lu et al. 2018); the paper made the continuous model trainable at scale and showed what it buys: adaptive computation, constant-memory backprop and a cheaper kind of normalizing flow.

A ResNet is Euler's method

A residual block adds a learned update to its input. Write the update with a step size Δt\Delta t and it is exactly one explicit Euler step of an ODE:

hk+1=hk+Δt fθ(hk,tk)→  Δt→0  dh(t)dt=fθ(h(t),t),h(t1)=h(t0)+∫t0t1fθ(h(t),t) dth_{k+1}=h_k+\Delta t\,f_\theta(h_k,t_k)\quad\xrightarrow{\;\Delta t\to0\;}\quad\frac{dh(t)}{dt}=f_\theta\big(h(t),t\big),\qquad h(t_1)=h(t_0)+\int_{t_0}^{t_1}f_\theta\big(h(t),t\big)\,dt
hkh_k
activation after block kk; in the limit, the state h(t)h(t) at "depth" tt
fθf_\theta
one network shared across all depths, with the time tt as an optional input
t0, t1t_0,\,t_1
start and end "times" (e.g. 0 and 1); h(t0)=xh(t_0)=x is the input

Depth is no longer a count of layers but a length of time, and the forward pass is h1 = ODESolve(f, h0, t0, t1). Any solver works. A ResNet with NN weight-tied blocks is the crudest choice, fixed-step Euler with Δt=1/N\Delta t=1/N. Higher-order and adaptive solvers reach the same h(t1)h(t_1) far more accurately per evaluation of ff.

The forward pass: adaptive solvers

An adaptive solver such as dopri5 (Dormand–Prince 5(4), the default in torchdiffeq) takes a step, estimates its local error by comparing a 5th- and a 4th-order answer computed from one shared set of evaluations of ff (six per step), and shrinks or grows the next step to keep that error within atol + rtol·|h|. The number of function evaluations (NFE) is then the model's depth, chosen per input.

This gives a speed–accuracy dial at test time: loosen the tolerance and the same trained model runs with fewer evaluations. The flip side, observed in the original paper, is that NFE tends to grow during training as fθf_\theta learns more complex dynamics, so a Neural ODE can become slower than the ResNet it replaced.

dh/dt = f(h) · t = 1.00 · Euler = ResNet with 4 blockssolver explicit Eulersteps N 4 (Δt = 0.250)evaluations of f 4mean error at t = 1 1.393dashed = exact flow (closed form)dots = solver steps = residual blocksmean error at t = 1 vs NFE (log–log)11e-21e-41e-61e-8141664128Euler RK4
field|
solver|
fig.Points flow along a 2D field from t = 0 to t = 1, solved live. Dashed: the exact flow. On the linear spiral, Euler with N = 4 steps (a 4-block ResNet) misses by 1.39 on average and still misses by 0.117 at N = 32, while RK4 at the same 32 evaluations is off by only 4.1e-4 (the chart shows error falling as 1/N for Euler and 1/N⁴ for RK4). Now pick 'push-out bump' and 'disk inside ring': the field shoves the centre right, but the exact flow can only drag the ring along. With one big Euler step all 15 inner points jump out of the ring (0/15 enclosed); from N = 4 on they stay inside.
steps NEuler h(1)Euler errorRK4 h(1)RK4 errorNFE (Euler / RK4)
12.00000.722.7083339.9×10−39.9\times10^{-3}1 / 4
42.44140.282.7182107.2×10−57.2\times10^{-5}4 / 16
102.59370.132.7182802.1×10−62.1\times10^{-6}10 / 40
1002.70480.0132.7182822.3×10−102.3\times10^{-10}100 / 400
Worked example: dh/dt = h, h(0) = 1, so h(1) = e = 2.718282. Euler with N steps is (1 + 1/N)ᴺ, the compound-interest formula.

At equal cost RK4 wins by orders of magnitude: 16 evaluations give RK4 an error of 7.2×10−57.2\times10^{-5}, while Euler with 16 steps is still off by 0.0800.080. Euler's error shrinks in proportion to 1/N1/N (first order), RK4's in proportion to 1/N41/N^4 (fourth order), which is why the RK4 line in the figure's chart is so much steeper.

The backward pass: the adjoint method

Backpropagating through the solver's internal operations works but stores every intermediate state, so memory grows with NFE. The adjoint sensitivity method (Pontryagin, 1962) instead tracks how the loss depends on the state at each time:

a(t)≡∂L∂h(t),da(t)dt=−a(t)⊤∂fθ(h(t),t)∂h,dLdθ=∫t0t1a(t)⊤∂fθ(h(t),t)∂θ dta(t)\equiv\frac{\partial\mathcal{L}}{\partial h(t)},\qquad \frac{da(t)}{dt}=-a(t)^{\top}\frac{\partial f_\theta(h(t),t)}{\partial h},\qquad \frac{d\mathcal{L}}{d\theta}=\int_{t_0}^{t_1}a(t)^{\top}\frac{\partial f_\theta(h(t),t)}{\partial\theta}\,dt
a(t)a(t)
the adjoint: the continuous analogue of backprop's error signal δ\delta; starts at a(t1)=∂L/∂h(t1)a(t_1)=\partial\mathcal{L}/\partial h(t_1)
a⊤∂f/∂ha^\top\partial f/\partial h
a vector–Jacobian product, one reverse-mode autodiff call through f
dL/dθd\mathcal{L}/d\theta
accumulated along the way; the paper writes it as −∫t1t0-\int_{t_1}^{t_0}, the same integral run backward

One backward pass with the adjoint method

  1. 1Forward: solve hh from t0t_0 to t1t_1 and keep only h(t1)h(t_1). Compute L\mathcal{L} and a(t1)=∂L/∂h(t1)a(t_1)=\partial\mathcal{L}/\partial h(t_1).
  2. 2Build the augmented state [ h,  a,  g ][\,h,\;a,\;g\,] with g(t1)=0g(t_1)=0, where gg will accumulate dL/dθd\mathcal{L}/d\theta.
  3. 3Solve it backward from t1t_1 to t0t_0: h˙=fθ(h,t)\dot h=f_\theta(h,t) recomputes the trajectory in reverse, a˙=−a⊤∂f/∂h\dot a=-a^\top\partial f/\partial h, and g˙=−a⊤∂f/∂θ\dot g=-a^\top\partial f/\partial\theta. Each step needs one vector–Jacobian product of ff.
  4. 4Read off ∂L/∂h(t0)=a(t0)\partial\mathcal{L}/\partial h(t_0)=a(t_0) (the gradient for the layers before) and dL/dθ=g(t0)d\mathcal{L}/d\theta=g(t_0).

Check it on h˙=θh\dot h=\theta h, h(0)=h0h(0)=h_0, L=h(1)\mathcal{L}=h(1), so L=h0eθ\mathcal{L}=h_0e^{\theta}. The adjoint ODE a˙=−θa\dot a=-\theta a with a(1)=1a(1)=1 gives a(t)=eθ(1−t)a(t)=e^{\theta(1-t)}, hence ∂L/∂h0=a(0)=eθ\partial\mathcal{L}/\partial h_0=a(0)=e^\theta. And dL/dθ=∫01eθ(1−t)⋅h0eθt dt=h0eθd\mathcal{L}/d\theta=\int_0^1 e^{\theta(1-t)}\cdot h_0e^{\theta t}\,dt=h_0e^\theta, matching the direct derivative. With h0=2h_0=2, θ=0.5\theta=0.5: L=3.2974\mathcal{L}=3.2974, ∂L/∂h0=1.6487\partial\mathcal{L}/\partial h_0=1.6487, dL/dθ=3.2974d\mathcal{L}/d\theta=3.2974.

Continuous normalizing flows

A normalizing flow needs log⁡p\log p of a transformed sample, which for a discrete invertible layer means a log-determinant of its Jacobian, O(d3)O(d^3) in general. In continuous time the change of variables becomes a trace:

∂log⁡p(z(t))∂t=−tr⁡ ⁣(∂fθ(z(t),t)∂z(t))\frac{\partial\log p\big(z(t)\big)}{\partial t}=-\operatorname{tr}\!\Big(\frac{\partial f_\theta(z(t),t)}{\partial z(t)}\Big)
z(t)z(t)
sample being transported from a simple base density (e.g. Gaussian) at t0t_0 to data at t1t_1
tr⁡(∂f/∂z)\operatorname{tr}(\partial f/\partial z)
divergence of the field: how fast volume expands; integrated alongside zz

Any network fθf_\theta is allowed (no triangular structure needed), and the trace costs O(d)O(d) backward passes exactly, or one with Hutchinson's stochastic estimator tr⁡(J)≈ϵ⊤Jϵ\operatorname{tr}(J)\approx\epsilon^\top J\epsilon, as in FFJORD (Grathwohl et al., 2019).

The limitation: trajectories cannot cross

When fθf_\theta is Lipschitz, solutions are unique, so two trajectories can never pass through the same point at the same time. The map x↦h(t1)x\mapsto h(t_1) is therefore a homeomorphism: it bends and stretches space but cannot tear it or pass one region through another. In 1D it cannot send 1↦−11\mapsto-1 and −1↦1-1\mapsto1; in 2D it cannot pull a disk out of the ring around it, so concentric circles never become linearly separable.

Augmented Neural ODEs (Dupont, Doucet & Teh, NeurIPS 2019) fix this by appending zeros, [x, 0]∈Rd+p[x,\,0]\in\mathbb{R}^{d+p}, and flowing in the larger space, where the inner points can go "over" the ring. That also gave smoother learned flows and lower NFE in their experiments. A discrete ResNet is not bound by the rule at all: a large enough step can jump one point past another, as the figure's single Euler step shows.

python
1import torch, torch.nn as nn
2from torchdiffeq import odeint_adjoint as odeint # O(1)-memory backward pass
3
4class ODEFunc(nn.Module):
5 def __init__(self, dim, hidden=64):
6 super().__init__()
7 self.net = nn.Sequential(nn.Linear(dim, hidden), nn.Tanh(), nn.Linear(hidden, dim))
8 self.nfe = 0
9 def forward(self, t, h): # torchdiffeq calls f(t, h)
10 self.nfe += 1
11 return self.net(h)
12
13class ODEBlock(nn.Module):
14 def __init__(self, dim, augment=0):
15 super().__init__()
16 self.augment, self.f = augment, ODEFunc(dim + augment)
17 self.t = torch.tensor([0.0, 1.0])
18 def forward(self, x):
19 h0 = torch.cat([x, x.new_zeros(x.shape[0], self.augment)], 1) # ANODE: pad with zeros
20 return odeint(self.f, h0, self.t, method="dopri5", rtol=1e-3, atol=1e-4)[-1]
21
22# The ResNet it generalises: N weight-tied Euler steps of the same f
23def resnet(f, h, N):
24 for k in range(N):
25 h = h + (1 / N) * f(k / N, h)
26 return h
An ODE block with torchdiffeq: dopri5 forward, adjoint backward, a function-evaluation counter, and the augmented variant

Use when

  • +Constant memory in depth with the adjoint method.
  • +Adaptive computation: the solver picks the depth; accuracy can be traded for speed at test time.
  • +Natural for continuous or irregularly sampled time and for flows (trace instead of log-determinant).

Avoid when

  • −Slow: sequential solver steps, and NFE tends to grow during training.
  • −Flows are homeomorphisms: no crossing trajectories without augmentation.
  • −Adjoint gradients can be inaccurate for stiff or dissipative dynamics; stiff fields need implicit solvers.

Check yourself

A weight-tied ResNet with 10 blocks h ← h + 0.1·f(h) is equivalent to which ODE solve of dh/dt = f(h) from t = 0 to 1?

Check yourself

Why can a (non-augmented) Neural ODE in 2D not map a disk and the ring around it to two linearly separable sets?

Remember

  • ✓ResNet hk+1=hk+Δt f(hk)h_{k+1}=h_k+\Delta t\,f(h_k) is Euler's method; Neural ODE: dh/dt=fθ(h,t)dh/dt=f_\theta(h,t), output =ODESolve(fθ,x,t0,t1)=\text{ODESolve}(f_\theta,x,t_0,t_1).
  • ✓Adaptive solvers (dopri5) choose the step size to meet a tolerance; NFE is the effective depth.
  • ✓Adjoint: a=∂L/∂ha=\partial\mathcal{L}/\partial h, a˙=−a⊤∂f/∂h\dot a=-a^\top\partial f/\partial h, dL/dθ=∫a⊤∂f/∂θ dtd\mathcal{L}/d\theta=\int a^\top\partial f/\partial\theta\,dt, solved backward for O(1)O(1) memory.
  • ✓CNF: dlog⁡p/dt=−tr⁡(∂f/∂z)d\log p/dt=-\operatorname{tr}(\partial f/\partial z); FFJORD estimates the trace stochastically.
  • ✓Trajectories cannot cross (flows are homeomorphisms); Augmented Neural ODEs add dimensions to escape this.
← back to Deep Learning roadmap