Liquid neural network

Continuous-time recurrent neurons whose time constant depends on the input, so the dynamics speed up or slow down with what they see while the state stays bounded.

9 min read760 wordsadvanced0 viewsupdated 2026-10-11

Definition

Liquid time-constant (LTC) network is a continuous-time recurrent network in which each neuron's state follows an ODE whose decay rate is modulated by a learned, input-dependent gate. Its effective time constant therefore changes from moment to moment ("liquid") with the input, unlike an RNN or a leaky integrator with a fixed time constant.

LTCs were introduced by Ramin Hasani, Mathias Lechner, Alexander Amini, Daniela Rus and Radu Grosu (arXiv 2020, AAAI 2021). The design borrows from models of non-spiking neurons in the nematode C. elegans, where a synapse acts like a conductance that pulls the membrane potential toward a reversal potential. "Liquid neural network" is the umbrella name for this family: LTCs, their wiring-constrained Neural Circuit Policies, and the closed-form CfC variant.

From a leaky integrator to a liquid one

A continuous-time RNN neuron is a leaky integrator: it decays toward zero at a fixed rate 1/τ1/\tau and is driven by its input, x˙=−x/τ+f(x,I;θ)\dot x=-x/\tau+f(x,I;\theta). Its speed of response is set once by τ\tau, and nothing stops xx from growing with the input.

An LTC neuron instead lets the drive enter multiplicatively, as a conductance that both opens a channel toward a target value AA and adds to the leak:

dx(t)dt=−[1τ+f(x(t),I(t);θ)] x(t)+f(x(t),I(t);θ) A\frac{dx(t)}{dt}=-\Big[\frac{1}{\tau}+f\big(x(t),I(t);\theta\big)\Big]\,x(t)+f\big(x(t),I(t);\theta\big)\,A
x(t)x(t)
hidden state of the neuron (its "membrane potential")
I(t)I(t)
input at time t
τ\tau
base time constant, learned and positive
f(⋅)≥0f(\cdot)\ge0
a small network with a sigmoid output (the synaptic conductance), parameters θ\theta
AA
learned bias vector (the reversal potential): where the gate pulls the state

Collect the terms that multiply xx and the equation reads x˙=−x/τsys+fA\dot x=-x/\tau_{\text{sys}}+fA with an input-dependent time constant:

τsys=τ1+τ f(x,I;θ),x∗=f A1/τ+f\tau_{\text{sys}}=\frac{\tau}{1+\tau\,f(x,I;\theta)},\qquad x^\ast=\frac{f\,A}{1/\tau+f}
τsys\tau_{\text{sys}}
effective time constant: equal to τ\tau when the gate is closed (f=0f=0), shorter as ff grows
x∗x^\ast
the value the state relaxes toward for a fixed input; always between 0 and AA

Two properties follow, and Hasani et al. prove both. If 0≤f≤g0\le f\le g, the time constant stays in [τ/(1+τg), τ][\tau/(1+\tau g),\,\tau], so the neuron never becomes arbitrarily fast or slow. And starting inside the range, the state stays between min⁡(0,A)\min(0,A) and max⁡(0,A)\max(0,A) however large the input: a strong input just opens the gate wider and pulls harder toward AA, it cannot push past it.

Solving it: the fused semi-implicit Euler step

The ODE is stiff when ff is large (a fast pull toward AA), and explicit Euler then overshoots or blows up. The LTC paper treats the decaying xx implicitly and the gate explicitly, which gives a closed-form update:

x(t+Δt)=x(t)+Δt f(x(t),I(t)) A1+Δt (1/τ+f(x(t),I(t)))x(t+\Delta t)=\frac{x(t)+\Delta t\,f\big(x(t),I(t)\big)\,A}{1+\Delta t\,\Big(1/\tau+f\big(x(t),I(t)\big)\Big)}
Δt\Delta t
solver step; the paper unfolds 6 steps per input sample

The result is a convex combination of x(t)x(t) and AA (both coefficients are positive), so the update itself respects the bound at any step size. Training is ordinary backpropagation through time through the unrolled solver steps.

05101520time t (τ = 1) →I(t)x(t)-101LTC bound: x stays between 0 and A = ±1τ_sys01■ LTC exc. (A = +1) ■ LTC inh. (A = −1) ■ leaky, fixed τ · dashed: fine-step reference
input|solver|
t = 20.00, I = 0.000exc x 0.267, τ_sys 0.733inh x -0.267, τ_sys 0.733leaky x 0.000 (gain 0.5, τ fixed 1)
fig.A real simulation of two LTC neurons, f(I) = 2σ(±3I − 1.5), τ = 1, next to a leaky integrator with fixed τ = 1. During the step the excitatory neuron's τ_sys drops from 0.733 to 0.379, so it rises faster; the inhibitory one's slows to 0.978. Push the amplitude to 3: the leaky integrator climbs to 1.5 while the LTC states stay inside ±1. Then switch to explicit Euler and drag Δt past 0.78: the excitatory neuron overshoots the bound, while the fused step stays stable at Δt = 1.2.
texactfused, Δt = 0.5explicit Euler, Δt = 0.5explicit Euler, Δt = 1
0.50.45440.35280.8176—
1.00.57600.50500.55791.6351
1.50.60860.57070.6404—
2.00.61730.59900.6142−1.0386
Worked example: the excitatory neuron from rest x = 0 with a constant input I = 1, so f = 2σ(1.5) = 1.6351, τ_sys = 0.3795 and x = 0.6205. Exact: x(t) = x(1 − e^(−t/τ_sys)).

Explicit Euler multiplies the distance to x∗x^\ast by 1−Δt/τsys1-\Delta t/\tau_{\text{sys}} each step. At Δt=0.5\Delta t=0.5 that is −0.32-0.32, so it oscillates around x∗x^\ast but converges; at Δt=1\Delta t=1 it is −1.64-1.64 and the iterates grow (the next two are 3.333.33 and −3.82-3.82). The fused step's factor is 1/(1+Δt/τsys)1/(1+\Delta t/\tau_{\text{sys}}), which lies in (0,1)(0,1) for every Δt\Delta t: it lags the exact curve slightly but never leaves [0,1][0,1].

python
1import torch, torch.nn as nn
2
3class LTCCell(nn.Module):
4 def __init__(self, n_in, n_hidden, unfolds=6):
5 super().__init__()
6 self.inp = nn.Linear(n_in, n_hidden)
7 self.rec = nn.Linear(n_hidden, n_hidden, bias=False)
8 self.log_tau = nn.Parameter(torch.zeros(n_hidden)) # tau = exp(.) > 0
9 self.A = nn.Parameter(torch.randn(n_hidden)) # reversal potentials
10 self.unfolds = unfolds
11
12 def forward(self, I, x, dt): # dt: (B, 1) elapsed time
13 tau, h = self.log_tau.exp(), dt / self.unfolds
14 for _ in range(self.unfolds):
15 f = torch.sigmoid(self.inp(I) + self.rec(x)) # conductance, in (0, 1)
16 x = (x + h * f * self.A) / (1 + h * (1 / tau + f)) # fused semi-implicit step
17 return x
18
19cell, head = LTCCell(3, 19), nn.Linear(19, 1)
20x = torch.zeros(8, 19)
21for I_t, dt_t in zip(inputs.unbind(1), gaps.unbind(1)): # (B, T, 3) inputs, (B, T, 1) time gaps
22 x = cell(I_t, x, dt_t)
23y_hat = head(x) # trained with ordinary BPTT
24
25# The authors' library: from ncps.torch import LTC, CfC; from ncps.wirings import AutoNCP
A dense LTC cell with the fused solver (the paper's f is built from per-synapse sigmoids; this is the same idea in one Linear layer). dt can differ per sample, which handles irregular sampling.

Neural Circuit Policies and CfC

Neural Circuit Policies (Lechner, Hasani et al., Nature Machine Intelligence 2020) wire LTC neurons sparsely in four layers modelled on the C. elegans tap-withdrawal circuit: sensory, inter, command and motor neurons. Behind a convolutional feature extractor, a 19-neuron NCP steered a car for lane keeping from camera images, comparable to much larger LSTM controllers, with attention maps that stayed on the road and a circuit small enough to inspect neuron by neuron. Later work (Chahine et al., Science Robotics 2023) used liquid networks to fly drones toward targets in environments unlike those seen in training.

The cost is the solver: every input sample needs several sequential update steps. Closed-form continuous-time networks (CfC, Hasani et al., Nature Machine Intelligence 2022) approximate the LTC solution in closed form and drop the solver entirely. The elapsed time tt enters as an input to a time-dependent gate:

x(t)=σ(−f(x,I;θf) t)⊙g(x,I;θg)+[1−σ(−f(x,I;θf) t)]⊙h(x,I;θh)x(t)=\sigma\big(-f(x,I;\theta_f)\,t\big)\odot g(x,I;\theta_g)+\Big[1-\sigma\big(-f(x,I;\theta_f)\,t\big)\Big]\odot h(x,I;\theta_h)
f, g, hf,\,g,\,h
three heads on a shared backbone network
σ(−f t)\sigma(-f\,t)
a learned, input-dependent interpolation from gg (short elapsed time) to hh (long elapsed time), playing the role of e−t/τsyse^{-t/\tau_{\text{sys}}}

The CfC paper reports training and inference one to five orders of magnitude faster than ODE-based counterparts, at similar accuracy on its time-series benchmarks.

Use when

  • +Input-dependent time constant: fast reaction to strong inputs, slow integration otherwise.
  • +Bounded state and bounded τsys\tau_{\text{sys}} by construction; the fused solver is stable at any step size.
  • +Elapsed time is explicit, so irregularly sampled series need no resampling.
  • +Very small, sparse, inspectable networks (NCP) for robot control.

Avoid when

  • −ODE unfolding per sample makes training and inference slow (CfC mitigates this).
  • −Sequential in time: no parallel training across the sequence like a transformer.
  • −Evidence is mostly small-scale; tooling and pretrained models are scarce.

Check yourself

An LTC neuron has τ = 2 and, for the current input, gate value f = 1.5. What is its effective time constant?

Check yourself

Why does a large input not push an LTC neuron's state beyond its reversal potential A?

Remember

  • ✓LTC: x˙=−[1/τ+f(x,I;θ)] x+f(x,I;θ) A\dot x=-[1/\tau+f(x,I;\theta)]\,x+f(x,I;\theta)\,A with f≥0f\ge0 a learned gate.
  • ✓Input-dependent time constant τsys=τ/(1+τf)\tau_{\text{sys}}=\tau/(1+\tau f), bounded in [τ/(1+τg),τ][\tau/(1+\tau g),\tau]; state bounded between 0 and AA.
  • ✓Fused semi-implicit Euler: x←(x+Δt fA)/(1+Δt(1/τ+f))x\leftarrow(x+\Delta t\,fA)/(1+\Delta t(1/\tau+f)), stable at any Δt\Delta t; explicit Euler is not.
  • ✓NCPs: sparse C. elegans-style wiring, 19 neurons steering a car; CfC (2022) replaces the solver with a closed-form approximation.
  • ✓Good for small, irregularly sampled, control-style problems; slow and unproven at large scale.
← back to Deep Learning roadmap