# Liquid neural network

> Continuous-time recurrent neurons whose time constant depends on the input, so the dynamics speed up or slow down with what they see while the state stays bounded.

Source: https://www.mlfoundry.in/learn/liquid-neural-network  
Section: Deep Learning › Neural Network Algorithms  
Level: advanced · 9 min read · updated 2026-10-11

**Liquid time-constant (LTC) network:** is a continuous-time recurrent network in which each neuron's state follows an ODE whose decay rate is modulated by a learned, input-dependent gate. Its effective time constant therefore changes from moment to moment ("liquid") with the input, unlike an RNN or a leaky integrator with a fixed time constant.

LTCs were introduced by Ramin Hasani, Mathias Lechner, Alexander Amini, Daniela Rus and Radu Grosu (arXiv 2020, AAAI 2021). The design borrows from models of non-spiking neurons in the nematode *C. elegans*, where a synapse acts like a conductance that pulls the membrane potential toward a reversal potential. "Liquid neural network" is the umbrella name for this family: LTCs, their wiring-constrained Neural Circuit Policies, and the closed-form CfC variant.

## From a leaky integrator to a liquid one

A continuous-time RNN neuron is a leaky integrator: it decays toward zero at a fixed rate $1/\tau$ and is driven by its input, $\dot x=-x/\tau+f(x,I;\theta)$. Its speed of response is set once by $\tau$, and nothing stops $x$ from growing with the input.

An LTC neuron instead lets the drive enter **multiplicatively**, as a conductance that both opens a channel toward a target value $A$ and adds to the leak:

$$
\frac{dx(t)}{dt}=-\Big[\frac{1}{\tau}+f\big(x(t),I(t);\theta\big)\Big]\,x(t)+f\big(x(t),I(t);\theta\big)\,A
$$

- $x(t)$: hidden state of the neuron (its "membrane potential")
- $I(t)$: input at time t
- $\tau$: base time constant, learned and positive
- $f(\cdot)\ge0$: a small network with a sigmoid output (the synaptic conductance), parameters $\theta$
- $A$: learned bias vector (the reversal potential): where the gate pulls the state

Collect the terms that multiply $x$ and the equation reads $\dot x=-x/\tau_{\text{sys}}+fA$ with an **input-dependent time constant**:

$$
\tau_{\text{sys}}=\frac{\tau}{1+\tau\,f(x,I;\theta)},\qquad x^\ast=\frac{f\,A}{1/\tau+f}
$$

- $\tau_{\text{sys}}$: effective time constant: equal to $\tau$ when the gate is closed ($f=0$), shorter as $f$ grows
- $x^\ast$: the value the state relaxes toward for a fixed input; always between 0 and $A$

Two properties follow, and Hasani et al. prove both. If $0\le f\le g$, the time constant stays in $[\tau/(1+\tau g),\,\tau]$, so the neuron never becomes arbitrarily fast or slow. And starting inside the range, the state stays between $\min(0,A)$ and $\max(0,A)$ however large the input: a strong input just opens the gate wider and pulls harder toward $A$, it cannot push past it.

## Solving it: the fused semi-implicit Euler step

The ODE is **stiff** when $f$ is large (a fast pull toward $A$), and explicit Euler then overshoots or blows up. The LTC paper treats the decaying $x$ implicitly and the gate explicitly, which gives a closed-form update:

$$
x(t+\Delta t)=\frac{x(t)+\Delta t\,f\big(x(t),I(t)\big)\,A}{1+\Delta t\,\Big(1/\tau+f\big(x(t),I(t)\big)\Big)}
$$

- $\Delta t$: solver step; the paper unfolds 6 steps per input sample

The result is a convex combination of $x(t)$ and $A$ (both coefficients are positive), so the update itself respects the bound at **any** step size. Training is ordinary backpropagation through time through the unrolled solver steps.

> Figure: A real simulation of two LTC neurons, f(I) = 2σ(±3I − 1.5), τ = 1, next to a leaky integrator with fixed τ = 1. During the step the excitatory neuron's τ_sys drops from 0.733 to 0.379, so it rises faster; the inhibitory one's slows to 0.978. Push the amplitude to 3: the leaky integrator climbs to 1.5 while the LTC states stay inside ±1. Then switch to explicit Euler and drag Δt past 0.78: the excitatory neuron overshoots the bound, while the fused step stays stable at Δt = 1.2.

Worked example: the excitatory neuron from rest x = 0 with a constant input I = 1, so f = 2σ(1.5) = 1.6351, τ_sys = 0.3795 and x* = 0.6205. Exact: x(t) = x*(1 − e^(−t/τ_sys)).

| t | exact | fused, Δt = 0.5 | explicit Euler, Δt = 0.5 | explicit Euler, Δt = 1 |
| --- | --- | --- | --- | --- |
| 0.5 | 0.4544 | 0.3528 | 0.8176 | — |
| 1.0 | 0.5760 | 0.5050 | 0.5579 | 1.6351 |
| 1.5 | 0.6086 | 0.5707 | 0.6404 | — |
| 2.0 | 0.6173 | 0.5990 | 0.6142 | −1.0386 |

Explicit Euler multiplies the distance to $x^\ast$ by $1-\Delta t/\tau_{\text{sys}}$ each step. At $\Delta t=0.5$ that is $-0.32$, so it oscillates around $x^\ast$ but converges; at $\Delta t=1$ it is $-1.64$ and the iterates grow (the next two are $3.33$ and $-3.82$). The fused step's factor is $1/(1+\Delta t/\tau_{\text{sys}})$, which lies in $(0,1)$ for every $\Delta t$: it lags the exact curve slightly but never leaves $[0,1]$.

```python
import torch, torch.nn as nn

class LTCCell(nn.Module):
    def __init__(self, n_in, n_hidden, unfolds=6):
        super().__init__()
        self.inp = nn.Linear(n_in, n_hidden)
        self.rec = nn.Linear(n_hidden, n_hidden, bias=False)
        self.log_tau = nn.Parameter(torch.zeros(n_hidden))    # tau = exp(.) > 0
        self.A = nn.Parameter(torch.randn(n_hidden))          # reversal potentials
        self.unfolds = unfolds

    def forward(self, I, x, dt):                              # dt: (B, 1) elapsed time
        tau, h = self.log_tau.exp(), dt / self.unfolds
        for _ in range(self.unfolds):
            f = torch.sigmoid(self.inp(I) + self.rec(x))      # conductance, in (0, 1)
            x = (x + h * f * self.A) / (1 + h * (1 / tau + f))   # fused semi-implicit step
        return x

cell, head = LTCCell(3, 19), nn.Linear(19, 1)
x = torch.zeros(8, 19)
for I_t, dt_t in zip(inputs.unbind(1), gaps.unbind(1)):  # (B, T, 3) inputs, (B, T, 1) time gaps
    x = cell(I_t, x, dt_t)
y_hat = head(x)                                           # trained with ordinary BPTT

# The authors' library: from ncps.torch import LTC, CfC; from ncps.wirings import AutoNCP
```

A dense LTC cell with the fused solver (the paper's f is built from per-synapse sigmoids; this is the same idea in one Linear layer). dt can differ per sample, which handles irregular sampling.

## Neural Circuit Policies and CfC

**Neural Circuit Policies** (Lechner, Hasani et al., *Nature Machine Intelligence* 2020) wire LTC neurons sparsely in four layers modelled on the *C. elegans* tap-withdrawal circuit: sensory, inter, command and motor neurons. Behind a convolutional feature extractor, a 19-neuron NCP steered a car for lane keeping from camera images, comparable to much larger LSTM controllers, with attention maps that stayed on the road and a circuit small enough to inspect neuron by neuron. Later work (Chahine et al., *Science Robotics* 2023) used liquid networks to fly drones toward targets in environments unlike those seen in training.

The cost is the solver: every input sample needs several sequential update steps. **Closed-form continuous-time networks** (CfC, Hasani et al., *Nature Machine Intelligence* 2022) approximate the LTC solution in closed form and drop the solver entirely. The elapsed time $t$ enters as an input to a time-dependent gate:

$$
x(t)=\sigma\big(-f(x,I;\theta_f)\,t\big)\odot g(x,I;\theta_g)+\Big[1-\sigma\big(-f(x,I;\theta_f)\,t\big)\Big]\odot h(x,I;\theta_h)
$$

- $f,\,g,\,h$: three heads on a shared backbone network
- $\sigma(-f\,t)$: a learned, input-dependent interpolation from $g$ (short elapsed time) to $h$ (long elapsed time), playing the role of $e^{-t/\tau_{\text{sys}}}$

The CfC paper reports training and inference one to five orders of magnitude faster than ODE-based counterparts, at similar accuracy on its time-series benchmarks.

> **Where the time constant lives:** An LSTM's forget gate also makes "how fast to forget" input-dependent, but per discrete step, with no notion of elapsed time. A Neural ODE has continuous time but no built-in leak or bound: $\dot x=f(x,I;\theta)$ can do anything $f$ can. LTCs sit in between: a fixed physical form (leak + conductance toward $A$) whose rate is learned and input-dependent, measured in real time units.

> **Not a drop-in sequence model at scale:** The published wins are on small control and time-series tasks (lane keeping, drone flight, physical and medical series). Solver-based LTCs are slow, sequential and still trained with BPTT, so very long sequences bring the usual vanishing-gradient problems. For large-scale language or vision they have not displaced transformers or state-space models.

**Strengths**

- Input-dependent time constant: fast reaction to strong inputs, slow integration otherwise.
- Bounded state and bounded $\tau_{\text{sys}}$ by construction; the fused solver is stable at any step size.
- Elapsed time is explicit, so irregularly sampled series need no resampling.
- Very small, sparse, inspectable networks (NCP) for robot control.

**Limitations**

- ODE unfolding per sample makes training and inference slow (CfC mitigates this).
- Sequential in time: no parallel training across the sequence like a transformer.
- Evidence is mostly small-scale; tooling and pretrained models are scarce.

**Question:** An LTC neuron has τ = 2 and, for the current input, gate value f = 1.5. What is its effective time constant?

- A. 0.5
- B. 2
- C. 0.67
- D. 3.5

**Answer:** A. 0.5 — $\tau_{\text{sys}}=\tau/(1+\tau f)=2/(1+3)=0.5$. Equivalently the total decay rate is $1/\tau+f=0.5+1.5=2$, whose inverse is $0.5$.

**Question:** Why does a large input not push an LTC neuron's state beyond its reversal potential A?

- A. The input enters as a conductance: a stronger input pulls harder toward A but also increases the leak
- B. The state is clipped to [−1, 1] after every step
- C. The sigmoid in f saturates the input at 1
- D. Because τ is learned to be large

**Answer:** A. The input enters as a conductance: a stronger input pulls harder toward A but also increases the leak — For a fixed input the state relaxes to $x^\ast=fA/(1/\tau+f)$, which approaches $A$ as $f\to\infty$ but never passes it. A leaky integrator's drive is additive instead, so its equilibrium grows linearly with the input, as the figure shows at amplitude 3.

**Key points**

- LTC: $\dot x=-[1/\tau+f(x,I;\theta)]\,x+f(x,I;\theta)\,A$ with $f\ge0$ a learned gate.
- Input-dependent time constant $\tau_{\text{sys}}=\tau/(1+\tau f)$, bounded in $[\tau/(1+\tau g),\tau]$; state bounded between 0 and $A$.
- Fused semi-implicit Euler: $x\leftarrow(x+\Delta t\,fA)/(1+\Delta t(1/\tau+f))$, stable at any $\Delta t$; explicit Euler is not.
- NCPs: sparse C. elegans-style wiring, 19 neurons steering a car; CfC (2022) replaces the solver with a closed-form approximation.
- Good for small, irregularly sampled, control-style problems; slow and unproven at large scale.

---

From MLFoundry (https://www.mlfoundry.in). The web page adds interactive figures and diagrams.
