# Neural ODE (Neural Ordinary Differential Equation)

> A ResNet is Euler's method; take the step to zero and the network becomes a learned vector field whose forward pass is an ODE solve, trained with constant memory by the adjoint method.

Source: https://www.mlfoundry.in/learn/neural-ode  
Section: Deep Learning › Neural Network Algorithms  
Level: advanced · 11 min read · updated 2026-10-11

**Neural ODE:** is a model whose hidden state evolves continuously according to $dh/dt=f_\theta(h(t),t)$, where $f_\theta$ is a neural network. The output is $h(t_1)$, obtained by running a numerical ODE solver from $h(t_0)=x$; gradients come from solving a second ODE backward in time (the adjoint method).

The idea was popularised by *Neural Ordinary Differential Equations* (Ricky T. Q. Chen, Yulia Rubanova, Jesse Bettencourt and David Duvenaud), a best-paper award winner at NeurIPS 2018. The view of residual networks as discretised dynamical systems was already around (E 2017; Haber & Ruthotto 2017; Lu et al. 2018); the paper made the continuous model trainable at scale and showed what it buys: adaptive computation, constant-memory backprop and a cheaper kind of normalizing flow.

## A ResNet is Euler's method

A residual block adds a learned update to its input. Write the update with a step size $\Delta t$ and it is exactly one explicit Euler step of an ODE:

$$
h_{k+1}=h_k+\Delta t\,f_\theta(h_k,t_k)\quad\xrightarrow{\;\Delta t\to0\;}\quad\frac{dh(t)}{dt}=f_\theta\big(h(t),t\big),\qquad h(t_1)=h(t_0)+\int_{t_0}^{t_1}f_\theta\big(h(t),t\big)\,dt
$$

- $h_k$: activation after block $k$; in the limit, the state $h(t)$ at "depth" $t$
- $f_\theta$: one network shared across all depths, with the time $t$ as an optional input
- $t_0,\,t_1$: start and end "times" (e.g. 0 and 1); $h(t_0)=x$ is the input

Depth is no longer a count of layers but a length of time, and the forward pass is `h1 = ODESolve(f, h0, t0, t1)`. Any solver works. A ResNet with $N$ weight-tied blocks is the crudest choice, fixed-step Euler with $\Delta t=1/N$. Higher-order and adaptive solvers reach the same $h(t_1)$ far more accurately per evaluation of $f$.

## The forward pass: adaptive solvers

An adaptive solver such as **dopri5** (Dormand–Prince 5(4), the default in `torchdiffeq`) takes a step, estimates its local error by comparing a 5th- and a 4th-order answer computed from one shared set of evaluations of $f$ (six per step), and shrinks or grows the next step to keep that error within `atol + rtol·|h|`. The number of function evaluations (**NFE**) is then the model's depth, chosen per input.

This gives a speed–accuracy dial at test time: loosen the tolerance and the same trained model runs with fewer evaluations. The flip side, observed in the original paper, is that NFE tends to **grow during training** as $f_\theta$ learns more complex dynamics, so a Neural ODE can become slower than the ResNet it replaced.

> Figure: Points flow along a 2D field from t = 0 to t = 1, solved live. Dashed: the exact flow. On the linear spiral, Euler with N = 4 steps (a 4-block ResNet) misses by 1.39 on average and still misses by 0.117 at N = 32, while RK4 at the same 32 evaluations is off by only 4.1e-4 (the chart shows error falling as 1/N for Euler and 1/N⁴ for RK4). Now pick 'push-out bump' and 'disk inside ring': the field shoves the centre right, but the exact flow can only drag the ring along. With one big Euler step all 15 inner points jump out of the ring (0/15 enclosed); from N = 4 on they stay inside.

Worked example: dh/dt = h, h(0) = 1, so h(1) = e = 2.718282. Euler with N steps is (1 + 1/N)ᴺ, the compound-interest formula.

| steps N | Euler h(1) | Euler error | RK4 h(1) | RK4 error | NFE (Euler / RK4) |
| --- | --- | --- | --- | --- | --- |
| 1 | 2.0000 | 0.72 | 2.708333 | $9.9\times10^{-3}$ | 1 / 4 |
| 4 | 2.4414 | 0.28 | 2.718210 | $7.2\times10^{-5}$ | 4 / 16 |
| 10 | 2.5937 | 0.13 | 2.718280 | $2.1\times10^{-6}$ | 10 / 40 |
| 100 | 2.7048 | 0.013 | 2.718282 | $2.3\times10^{-10}$ | 100 / 400 |

At equal cost RK4 wins by orders of magnitude: 16 evaluations give RK4 an error of $7.2\times10^{-5}$, while Euler with 16 steps is still off by $0.080$. Euler's error shrinks in proportion to $1/N$ (first order), RK4's in proportion to $1/N^4$ (fourth order), which is why the RK4 line in the figure's chart is so much steeper.

## The backward pass: the adjoint method

Backpropagating through the solver's internal operations works but stores every intermediate state, so memory grows with NFE. The adjoint sensitivity method (Pontryagin, 1962) instead tracks how the loss depends on the state at each time:

$$
a(t)\equiv\frac{\partial\mathcal{L}}{\partial h(t)},\qquad \frac{da(t)}{dt}=-a(t)^{\top}\frac{\partial f_\theta(h(t),t)}{\partial h},\qquad \frac{d\mathcal{L}}{d\theta}=\int_{t_0}^{t_1}a(t)^{\top}\frac{\partial f_\theta(h(t),t)}{\partial\theta}\,dt
$$

- $a(t)$: the adjoint: the continuous analogue of backprop's error signal $\delta$; starts at $a(t_1)=\partial\mathcal{L}/\partial h(t_1)$
- $a^\top\partial f/\partial h$: a vector–Jacobian product, one reverse-mode autodiff call through f
- $d\mathcal{L}/d\theta$: accumulated along the way; the paper writes it as $-\int_{t_1}^{t_0}$, the same integral run backward

**One backward pass with the adjoint method**

1. Forward: solve $h$ from $t_0$ to $t_1$ and keep **only** $h(t_1)$. Compute $\mathcal{L}$ and $a(t_1)=\partial\mathcal{L}/\partial h(t_1)$.
2. Build the augmented state $[\,h,\;a,\;g\,]$ with $g(t_1)=0$, where $g$ will accumulate $d\mathcal{L}/d\theta$.
3. Solve it **backward** from $t_1$ to $t_0$: $\dot h=f_\theta(h,t)$ recomputes the trajectory in reverse, $\dot a=-a^\top\partial f/\partial h$, and $\dot g=-a^\top\partial f/\partial\theta$. Each step needs one vector–Jacobian product of $f$.
4. Read off $\partial\mathcal{L}/\partial h(t_0)=a(t_0)$ (the gradient for the layers before) and $d\mathcal{L}/d\theta=g(t_0)$.

Check it on $\dot h=\theta h$, $h(0)=h_0$, $\mathcal{L}=h(1)$, so $\mathcal{L}=h_0e^{\theta}$. The adjoint ODE $\dot a=-\theta a$ with $a(1)=1$ gives $a(t)=e^{\theta(1-t)}$, hence $\partial\mathcal{L}/\partial h_0=a(0)=e^\theta$. And $d\mathcal{L}/d\theta=\int_0^1 e^{\theta(1-t)}\cdot h_0e^{\theta t}\,dt=h_0e^\theta$, matching the direct derivative. With $h_0=2$, $\theta=0.5$: $\mathcal{L}=3.2974$, $\partial\mathcal{L}/\partial h_0=1.6487$, $d\mathcal{L}/d\theta=3.2974$.

> **O(1) memory is not free:** The adjoint recomputes $h(t)$ by integrating backward, and for dissipative or stiff dynamics that reverse solve can drift from the forward trajectory, giving inaccurate gradients. The backward solve also usually costs more evaluations than the forward one. If memory allows, plain backprop through the solver (`odeint` instead of `odeint_adjoint`) is often faster and exact for the discretisation used; checkpointed variants sit in between.

## Continuous normalizing flows

A normalizing flow needs $\log p$ of a transformed sample, which for a discrete invertible layer means a log-determinant of its Jacobian, $O(d^3)$ in general. In continuous time the change of variables becomes a trace:

$$
\frac{\partial\log p\big(z(t)\big)}{\partial t}=-\operatorname{tr}\!\Big(\frac{\partial f_\theta(z(t),t)}{\partial z(t)}\Big)
$$

- $z(t)$: sample being transported from a simple base density (e.g. Gaussian) at $t_0$ to data at $t_1$
- $\operatorname{tr}(\partial f/\partial z)$: divergence of the field: how fast volume expands; integrated alongside $z$

Any network $f_\theta$ is allowed (no triangular structure needed), and the trace costs $O(d)$ backward passes exactly, or one with Hutchinson's stochastic estimator $\operatorname{tr}(J)\approx\epsilon^\top J\epsilon$, as in FFJORD (Grathwohl et al., 2019).

## The limitation: trajectories cannot cross

When $f_\theta$ is Lipschitz, solutions are unique, so two trajectories can never pass through the same point at the same time. The map $x\mapsto h(t_1)$ is therefore a **homeomorphism**: it bends and stretches space but cannot tear it or pass one region through another. In 1D it cannot send $1\mapsto-1$ and $-1\mapsto1$; in 2D it cannot pull a disk out of the ring around it, so concentric circles never become linearly separable.

**Augmented Neural ODEs** (Dupont, Doucet & Teh, NeurIPS 2019) fix this by appending zeros, $[x,\,0]\in\mathbb{R}^{d+p}$, and flowing in the larger space, where the inner points can go "over" the ring. That also gave smoother learned flows and lower NFE in their experiments. A discrete ResNet is not bound by the rule at all: a large enough step can jump one point past another, as the figure's single Euler step shows.

```python
import torch, torch.nn as nn
from torchdiffeq import odeint_adjoint as odeint      # O(1)-memory backward pass

class ODEFunc(nn.Module):
    def __init__(self, dim, hidden=64):
        super().__init__()
        self.net = nn.Sequential(nn.Linear(dim, hidden), nn.Tanh(), nn.Linear(hidden, dim))
        self.nfe = 0
    def forward(self, t, h):                           # torchdiffeq calls f(t, h)
        self.nfe += 1
        return self.net(h)

class ODEBlock(nn.Module):
    def __init__(self, dim, augment=0):
        super().__init__()
        self.augment, self.f = augment, ODEFunc(dim + augment)
        self.t = torch.tensor([0.0, 1.0])
    def forward(self, x):
        h0 = torch.cat([x, x.new_zeros(x.shape[0], self.augment)], 1)   # ANODE: pad with zeros
        return odeint(self.f, h0, self.t, method="dopri5", rtol=1e-3, atol=1e-4)[-1]

# The ResNet it generalises: N weight-tied Euler steps of the same f
def resnet(f, h, N):
    for k in range(N):
        h = h + (1 / N) * f(k / N, h)
    return h
```

An ODE block with torchdiffeq: dopri5 forward, adjoint backward, a function-evaluation counter, and the augmented variant

> **Where Neural ODEs earned their place:** Image classification was a proof of concept (on MNIST the ODE-Net matched a ResNet's 0.4% error with about a third of the parameters and constant memory). The lasting uses are where continuous time is the point: **latent ODEs** for irregularly sampled time series (Rubanova et al., 2019), continuous normalizing flows, physics-informed and scientific models, and as the conceptual ancestor of the probability-flow ODEs used to sample diffusion models and flow matching.

**Strengths**

- Constant memory in depth with the adjoint method.
- Adaptive computation: the solver picks the depth; accuracy can be traded for speed at test time.
- Natural for continuous or irregularly sampled time and for flows (trace instead of log-determinant).

**Limitations**

- Slow: sequential solver steps, and NFE tends to grow during training.
- Flows are homeomorphisms: no crossing trajectories without augmentation.
- Adjoint gradients can be inaccurate for stiff or dissipative dynamics; stiff fields need implicit solvers.

**Question:** A weight-tied ResNet with 10 blocks h ← h + 0.1·f(h) is equivalent to which ODE solve of dh/dt = f(h) from t = 0 to 1?

- A. Explicit Euler with 10 steps of size 0.1
- B. RK4 with 10 steps
- C. One step of an adaptive solver
- D. Euler with 1 step of size 10

**Answer:** A. Explicit Euler with 10 steps of size 0.1 — Each block is $h_{k+1}=h_k+\Delta t\,f(h_k)$ with $\Delta t=0.1$: one explicit Euler step. Ten of them cover $t\in[0,1]$. Its error shrinks only linearly with $\Delta t$; for $\dot h=h$ it gives $1.1^{10}=2.5937$ instead of $e$.

**Question:** Why can a (non-augmented) Neural ODE in 2D not map a disk and the ring around it to two linearly separable sets?

- A. Its flow is a homeomorphism, so trajectories cannot cross and the image of the ring always encloses the image of the disk
- B. The adjoint method cannot compute gradients for that loss
- C. Adaptive solvers refuse to take steps large enough
- D. tanh activations bound the state to [−1, 1]

**Answer:** A. Its flow is a homeomorphism, so trajectories cannot cross and the image of the ring always encloses the image of the disk — Unique solutions mean no two trajectories meet, so the continuous flow deforms the ring without ever letting a disk point pass through it. Augmenting with an extra dimension, or using a discrete ResNet with large steps, removes the constraint.

**Key points**

- ResNet $h_{k+1}=h_k+\Delta t\,f(h_k)$ is Euler's method; Neural ODE: $dh/dt=f_\theta(h,t)$, output $=\text{ODESolve}(f_\theta,x,t_0,t_1)$.
- Adaptive solvers (dopri5) choose the step size to meet a tolerance; NFE is the effective depth.
- Adjoint: $a=\partial\mathcal{L}/\partial h$, $\dot a=-a^\top\partial f/\partial h$, $d\mathcal{L}/d\theta=\int a^\top\partial f/\partial\theta\,dt$, solved backward for $O(1)$ memory.
- CNF: $d\log p/dt=-\operatorname{tr}(\partial f/\partial z)$; FFJORD estimates the trace stochastically.
- Trajectories cannot cross (flows are homeomorphisms); Augmented Neural ODEs add dimensions to escape this.

---

From MLFoundry (https://www.mlfoundry.in). The web page adds interactive figures and diagrams.
