Definition
Neural ODE is a model whose hidden state evolves continuously according to , where is a neural network. The output is , obtained by running a numerical ODE solver from ; gradients come from solving a second ODE backward in time (the adjoint method).
The idea was popularised by Neural Ordinary Differential Equations (Ricky T. Q. Chen, Yulia Rubanova, Jesse Bettencourt and David Duvenaud), a best-paper award winner at NeurIPS 2018. The view of residual networks as discretised dynamical systems was already around (E 2017; Haber & Ruthotto 2017; Lu et al. 2018); the paper made the continuous model trainable at scale and showed what it buys: adaptive computation, constant-memory backprop and a cheaper kind of normalizing flow.
A ResNet is Euler's method
A residual block adds a learned update to its input. Write the update with a step size and it is exactly one explicit Euler step of an ODE:
- activation after block ; in the limit, the state at "depth"
- one network shared across all depths, with the time as an optional input
- start and end "times" (e.g. 0 and 1); is the input
Depth is no longer a count of layers but a length of time, and the forward pass is h1 = ODESolve(f, h0, t0, t1). Any solver works. A ResNet with weight-tied blocks is the crudest choice, fixed-step Euler with . Higher-order and adaptive solvers reach the same far more accurately per evaluation of .
The forward pass: adaptive solvers
An adaptive solver such as dopri5 (Dormand–Prince 5(4), the default in torchdiffeq) takes a step, estimates its local error by comparing a 5th- and a 4th-order answer computed from one shared set of evaluations of (six per step), and shrinks or grows the next step to keep that error within atol + rtol·|h|. The number of function evaluations (NFE) is then the model's depth, chosen per input.
This gives a speed–accuracy dial at test time: loosen the tolerance and the same trained model runs with fewer evaluations. The flip side, observed in the original paper, is that NFE tends to grow during training as learns more complex dynamics, so a Neural ODE can become slower than the ResNet it replaced.
| steps N | Euler h(1) | Euler error | RK4 h(1) | RK4 error | NFE (Euler / RK4) |
|---|---|---|---|---|---|
| 1 | 2.0000 | 0.72 | 2.708333 | 1 / 4 | |
| 4 | 2.4414 | 0.28 | 2.718210 | 4 / 16 | |
| 10 | 2.5937 | 0.13 | 2.718280 | 10 / 40 | |
| 100 | 2.7048 | 0.013 | 2.718282 | 100 / 400 |
At equal cost RK4 wins by orders of magnitude: 16 evaluations give RK4 an error of , while Euler with 16 steps is still off by . Euler's error shrinks in proportion to (first order), RK4's in proportion to (fourth order), which is why the RK4 line in the figure's chart is so much steeper.
The backward pass: the adjoint method
Backpropagating through the solver's internal operations works but stores every intermediate state, so memory grows with NFE. The adjoint sensitivity method (Pontryagin, 1962) instead tracks how the loss depends on the state at each time:
- the adjoint: the continuous analogue of backprop's error signal ; starts at
- a vector–Jacobian product, one reverse-mode autodiff call through f
- accumulated along the way; the paper writes it as , the same integral run backward
One backward pass with the adjoint method
- 1Forward: solve from to and keep only . Compute and .
- 2Build the augmented state with , where will accumulate .
- 3Solve it backward from to : recomputes the trajectory in reverse, , and . Each step needs one vector–Jacobian product of .
- 4Read off (the gradient for the layers before) and .
Check it on , , , so . The adjoint ODE with gives , hence . And , matching the direct derivative. With , : , , .
Continuous normalizing flows
A normalizing flow needs of a transformed sample, which for a discrete invertible layer means a log-determinant of its Jacobian, in general. In continuous time the change of variables becomes a trace:
- sample being transported from a simple base density (e.g. Gaussian) at to data at
- divergence of the field: how fast volume expands; integrated alongside
Any network is allowed (no triangular structure needed), and the trace costs backward passes exactly, or one with Hutchinson's stochastic estimator , as in FFJORD (Grathwohl et al., 2019).
The limitation: trajectories cannot cross
When is Lipschitz, solutions are unique, so two trajectories can never pass through the same point at the same time. The map is therefore a homeomorphism: it bends and stretches space but cannot tear it or pass one region through another. In 1D it cannot send and ; in 2D it cannot pull a disk out of the ring around it, so concentric circles never become linearly separable.
Augmented Neural ODEs (Dupont, Doucet & Teh, NeurIPS 2019) fix this by appending zeros, , and flowing in the larger space, where the inner points can go "over" the ring. That also gave smoother learned flows and lower NFE in their experiments. A discrete ResNet is not bound by the rule at all: a large enough step can jump one point past another, as the figure's single Euler step shows.
1import torch, torch.nn as nn2from torchdiffeq import odeint_adjoint as odeint # O(1)-memory backward pass34class ODEFunc(nn.Module):5 def __init__(self, dim, hidden=64):6 super().__init__()7 self.net = nn.Sequential(nn.Linear(dim, hidden), nn.Tanh(), nn.Linear(hidden, dim))8 self.nfe = 09 def forward(self, t, h): # torchdiffeq calls f(t, h)10 self.nfe += 111 return self.net(h)1213class ODEBlock(nn.Module):14 def __init__(self, dim, augment=0):15 super().__init__()16 self.augment, self.f = augment, ODEFunc(dim + augment)17 self.t = torch.tensor([0.0, 1.0])18 def forward(self, x):19 h0 = torch.cat([x, x.new_zeros(x.shape[0], self.augment)], 1) # ANODE: pad with zeros20 return odeint(self.f, h0, self.t, method="dopri5", rtol=1e-3, atol=1e-4)[-1]2122# The ResNet it generalises: N weight-tied Euler steps of the same f23def resnet(f, h, N):24 for k in range(N):25 h = h + (1 / N) * f(k / N, h)26 return hUse when
- +Constant memory in depth with the adjoint method.
- +Adaptive computation: the solver picks the depth; accuracy can be traded for speed at test time.
- +Natural for continuous or irregularly sampled time and for flows (trace instead of log-determinant).
Avoid when
- −Slow: sequential solver steps, and NFE tends to grow during training.
- −Flows are homeomorphisms: no crossing trajectories without augmentation.
- −Adjoint gradients can be inaccurate for stiff or dissipative dynamics; stiff fields need implicit solvers.
Check yourself
A weight-tied ResNet with 10 blocks h ← h + 0.1·f(h) is equivalent to which ODE solve of dh/dt = f(h) from t = 0 to 1?
Check yourself
Why can a (non-augmented) Neural ODE in 2D not map a disk and the ring around it to two linearly separable sets?
Remember
- ✓ResNet is Euler's method; Neural ODE: , output .
- ✓Adaptive solvers (dopri5) choose the step size to meet a tolerance; NFE is the effective depth.
- ✓Adjoint: , , , solved backward for memory.
- ✓CNF: ; FFJORD estimates the trace stochastically.
- ✓Trajectories cannot cross (flows are homeomorphisms); Augmented Neural ODEs add dimensions to escape this.