Echo state network

A large, fixed, random recurrent reservoir turns the input history into a rich state, and only a linear readout is trained, by ridge regression in closed form. It works when the reservoir forgets its initial state, typically with a spectral radius near 1.

12 min read1094 wordsadvanced0 viewsupdated 2026-10-12

Definition

Echo state network (ESN; Jaeger, 2001) is a recurrent network whose recurrent and input weights are drawn at random and never trained. The recurrent layer, called the reservoir, is driven by the input and acts as a nonlinear fading memory of it; the output is a linear readout y(n)=Wout[1;u(n);x(n)]y(n)=W_{out}[1;u(n);x(n)] fitted by ridge regression. Training an RNN is reduced to one linear least-squares problem.

In 2001, training RNNs by backpropagation through time was slow and fragile: gradients vanished or exploded (Bengio et al., 1994), and bifurcations in the dynamics made the loss jump. Herbert Jaeger's GMD report "The 'echo state' approach to analysing and training recurrent neural networks" sidestepped the problem: if the random dynamics already contain the right features of the input history, there is nothing to backpropagate through. Independently, Maass, Natschläger and Markram (2002) proposed the liquid state machine, the same idea with spiking neurons as a model of cortical microcircuits. The two are now grouped as reservoir computing.

The approach became famous with Jaeger and Haas (Science, 2004), who used a 1000-unit ESN on the chaotic Mackey–Glass benchmark and reported that "accuracy is improved by a factor of 2400 over previous techniques", and improved the signal error rate of a nonlinear channel equaliser by two orders of magnitude. Reservoirs remain popular for chaotic time series, for very cheap online learning, and as physical computers (photonic, spintronic, mechanical) where the "reservoir" is a physical system that cannot be trained at all.

The model

x~(n)=tanh⁡ ⁣(Win[1;u(n)]+W x(n−1)),x(n)=(1−a) x(n−1)+a x~(n),y^(n)=Wout [1;u(n);x(n)]\tilde x(n)=\tanh\!\big(W_{in}[1;u(n)]+W\,x(n-1)\big),\qquad x(n)=(1-a)\,x(n-1)+a\,\tilde x(n),\qquad \hat y(n)=W_{out}\,[1;u(n);x(n)]
x(n)∈RNx(n)\in\mathbb R^{N}
reservoir state; NN is typically 10210^2–10410^4
WW
sparse random N×NN\times N recurrent matrix, rescaled to a chosen spectral radius ρ(W)\rho(W); fixed
WinW_{in}
random input weights (and bias), uniform in [−s,s][-s,s]; the input scaling ss sets how nonlinear the units are; fixed
a∈(0,1]a\in(0,1]
leak rate (Jaeger et al., 2007, leaky-integrator ESN). a=1a=1 is the plain ESN; small aa slows the dynamics to match slow inputs
WoutW_{out}
the only trained parameters: Dy×(1+Du+N)D_y\times(1+D_u+N)

Training collects the states of one long run into a matrix XX (one column [1;u(n);x(n)][1;u(n);x(n)] per step, after discarding a washout while the arbitrary initial state fades) and the targets into YY, then solves the regularised normal equations. Because nothing upstream of WoutW_{out} changes, this is a convex problem with a unique solution, and its cost is one pass of the reservoir plus a (1+Du+N)(1+D_u+N)-sized linear solve.

Wout=YX⊤(XX⊤+βI)−1W_{out}=Y X^\top\big(XX^\top+\beta I\big)^{-1}
XX
(1+Du+N)×Ttrain(1+D_u+N)\times T_{train} matrix of collected features
β\beta
ridge (Tikhonov) strength; keeps ∥Wout∥\lVert W_{out}\rVert small so the readout does not amplify tiny state differences

The echo state property and the spectral radius

An ESN only works if its state is a function of the input history alone: two copies of the reservoir started from different states and fed the same input must converge. Jaeger called this the echo state property (ESP); the state is an "echo" of the input. Without it the readout would be fitting the accident of the initial condition, and with too much of it the reservoir would forget the input before the readout can use it.

ρ(W)=max⁡k∣λk(W)∣<1(rule of thumb),σmax⁡(W)<1  ⇒  ESP for every input,leaky: ∣λk∣eff=∣(1−a)+aλk∣≤(1−a)+aρ\rho(W)=\max_k|\lambda_k(W)|<1\quad(\text{rule of thumb}),\qquad \sigma_{\max}(W)<1\;\Rightarrow\;\text{ESP for every input},\qquad \text{leaky: }|\lambda_k|_{\text{eff}}=|(1-a)+a\lambda_k|\le(1-a)+a\rho
ρ(W)\rho(W)
spectral radius: the asymptotic per-step growth of the linearised dynamics around x=0x=0
σmax⁡(W)\sigma_{\max}(W)
largest singular value; <1<1 makes the tanh update a contraction (Jaeger's sufficient condition, very conservative)
(1−a)+aλ(1-a)+a\lambda
eigenvalues of the leaky update's linear part (1−a)I+aW(1-a)I+aW

ρ(W)<1\rho(W)<1 is neither necessary nor sufficient in general: tanh saturation driven by a strong input can restore contraction when ρ>1\rho>1, and non-normal matrices can amplify transients even when ρ<1\rho<1. It is still the knob everyone tunes, because it sets the memory time scale: the closer the effective radius is to 1, the slower the echo of an input fades. Lukoševičius (2012) recommends treating ρ\rho, the input scaling and the leak rate as the three main hyperparameters, with ρ\rho near 1 for tasks that need long memory.

At a glance

Trained
only WoutW_{out}, in closed form (ridge regression); no BPTT, no gradients
Fixed
WW (sparse, rescaled to spectral radius ρ\rho), WinW_{in}, bias
Key hyperparameters
ρ(W)\rho(W), input scaling, leak rate aa, ridge β\beta, reservoir size NN
Condition
echo state property: the initial state is forgotten; ρ<1\rho<1 as rule of thumb
Cost
one reservoir run, O(T N⋅nnz per row)O(T\,N\cdot\text{nnz per row}), plus one (N+1+Du)(N+1+D_u)-dim linear solve
Good at
chaotic and dynamical time series, fast online adaptation, physical/neuromorphic computing
Relatives
liquid state machine (Maass et al., 2002); Liquid neural network and Neural ODE are trained continuous-time cousins
predict u(n+50) on unseen data · test NRMSE 0.0624 · train 0.0528 · input-only linear 0.725utest steps n = 1300 … 1599━ readout ŷ ┄ target (Mackey–Glass, τ = 17)4 of 100 reservoir units xᵢ(n) (never trained)echo state check: ‖x_a(n) − x_b(n)‖, same input11e-41e-81e-12after 200 steps: 3.4e-8 · initial state forgottenspectral radius ρ(W)0.90leak a0.30(1 − a) + a·ρ, bound0.970ridge β1e-6‖W_out‖949.6
horizon h:100 units, 1200 training steps, readout by ridge regression
fig.A 100-unit ESN (10% sparse, tanh, leak rate a) driven by the Mackey–Glass series (τ = 17), with a ridge readout trained on 1200 steps to predict the value h steps ahead and tested on the next 400. The random W has spectral radius 3.37 and is rescaled to ρ. At the defaults ρ = 0.9, a = 0.3, β = 1e−6, h = 50, test NRMSE is 0.0624 (train 0.0528), while the best linear predictor from u(n) alone gets 0.725. Slide ρ: 0.1 gives 0.312 (the reservoir forgets too fast), 1.2 gives 0.0425, 2.0 gives 0.205. The lower-right panel runs the reservoir from two random states with the same input: at ρ = 0.9 their distance falls from 6.0 to 3e−8 in 200 steps (echo state property); at ρ = 2.0 it is still 2.3. Without leakage (a = 1) the default ρ gives 0.114.

Reading the figure: memory versus stability

The task needs memory: to predict 50 steps ahead the readout must see the recent shape of the curve, not just its current value, which is why the input-only predictor fails (NRMSE 0.725). A small ρ\rho makes every echo vanish within a few steps, so the state carries little more than u(n)u(n). Raising ρ\rho lengthens the memory and the error drops by more than a factor of 7 up to around ρ≈1.2\rho\approx1.2 with a=0.3a=0.3, whose effective radius bound is 0.7+0.3⋅1.2=1.060.7+0.3\cdot1.2=1.06. Beyond that, the reservoir starts to generate its own dynamics and loses the echo state property: at ρ=2\rho=2 two initial states never merge, and the readout is fitting noise of its own making.

settingtest NRMSEtrain NRMSE‖W_out‖
ρ=0.1\rho=0.1, a=0.3a=0.30.31220.33776328.6
ρ=0.5\rho=0.5, a=0.3a=0.30.14380.12632016.2
ρ=0.9\rho=0.9, a=0.3a=0.3 (default)0.06240.0528949.6
ρ=1.2\rho=1.2, a=0.3a=0.30.04250.0346395.1
ρ=2.0\rho=2.0, a=0.3a=0.30.20460.165718.0
ρ=0.9\rho=0.9, a=1a=1 (no leak)0.11410.11102100.0
ρ=0.9\rho=0.9, a=0.3a=0.3, β=1\beta=10.47190.44418.1
Test NRMSE for the 50-step-ahead task (figure's lib, run in Node; β = 1e−6 unless stated)

The leak rate works on time scale: Mackey–Glass varies slowly relative to one sample, and a=0.3a=0.3 makes each unit a low-pass filter, which beat the plain ESN (a=1a=1) by almost a factor of 2 at the same ρ\rho. Ridge matters for a different reason: on this noise-free series a tiny β\beta is best for one-step evaluation (1e−10 gives 0.0463), but the readout norm grows to 7727, and a readout that large is fragile when the ESN is run in closed loop as a generator, feeding its prediction back as the next input. Tune β\beta on the task you will actually run.

Worked example: an echo fading in a 2-unit reservoir (checked in Node)

  1. 1Take W=(00.8−0.80)W=\begin{pmatrix}0&0.8\\-0.8&0\end{pmatrix} (eigenvalues ±0.8i\pm0.8i, so ρ=0.8\rho=0.8), Win=(1,0)⊤W_{in}=(1,0)^\top, no bias, a=1a=1, x(0)=0x(0)=0, and one input pulse u=(1,0,0,0,0)u=(1,0,0,0,0).
  2. 2x(1)=tanh⁡(1,0)=(0.7616, 0)x(1)=\tanh(1,0)=(0.7616,\,0). Then x(2)=tanh⁡(Wx(1))=tanh⁡(0,−0.6093)=(0, −0.5436)x(2)=\tanh(Wx(1))=\tanh(0,-0.6093)=(0,\,-0.5436), x(3)=(−0.4094, 0)x(3)=(-0.4094,\,0), x(4)=(0, 0.3163)x(4)=(0,\,0.3163), x(5)=(0.2478, 0)x(5)=(0.2478,\,0).
  3. 3The norms 0.762,0.544,0.409,0.316,0.2480.762, 0.544, 0.409, 0.316, 0.248 shrink by 0.714,0.753,0.773,0.7830.714, 0.753, 0.773, 0.783 per step, approaching ρ=0.8\rho=0.8 as the state leaves tanh's curved region. The pulse is remembered, rotating between the units, and fades geometrically: an echo.
  4. 4With leak a=0.5a=0.5 the effective eigenvalues are 0.5±0.4i0.5\pm0.4i, of modulus 0.6400.640, and the simulated norms 0.381,0.241,0.154,0.098,0.0630.381, 0.241, 0.154, 0.098, 0.063 shrink by 0.630.63–0.640.64 per step. The bound (1−a)+aρ=0.9(1-a)+a\rho=0.9 is loose here because the eigenvalues are imaginary.
python
1import torch
2torch.set_default_dtype(torch.float64) # ridge with tiny beta needs float64
3g = torch.Generator().manual_seed(0)
4
5def mackey_glass(n, tau=17, dt=0.1, sub=10, discard=500):
6 lag, x, hist, out = int(tau / dt), 1.2, [1.2] * int(tau / dt), []
7 for s in range((n + discard) * sub):
8 xl = hist[s % lag] # x(t - tau)
9 hist[s % lag] = x
10 x = x + dt * (0.2 * xl / (1 + xl ** 10) - 0.1 * x)
11 if (s + 1) % sub == 0 and (s + 1) // sub > discard:
12 out.append(x)
13 return torch.tensor(out)
14
15N, rho, leak, beta, h = 100, 0.9, 0.3, 1e-6, 50
16W = torch.randn(N, N, generator=g) * (torch.rand(N, N, generator=g) < 0.1)
17W *= rho / torch.linalg.eigvals(W).abs().max() # set the spectral radius
18W_in = (torch.rand(N, 2, generator=g) * 2 - 1) * 0.5 # [bias, input] columns
19
20@torch.no_grad()
21def run(u): # u: (T,) -> states (T, N)
22 x, xs = torch.zeros(N), []
23 for ut in u:
24 x_tilde = torch.tanh(W_in @ torch.stack([torch.tensor(1.0), ut]) + W @ x)
25 x = (1 - leak) * x + leak * x_tilde
26 xs.append(x)
27 return torch.stack(xs)
28
29u = mackey_glass(1900)
30u = (u - u.mean()) * 4
31X = run(u)
32feats = torch.cat([torch.ones(len(u), 1), u[:, None], X], dim=1) # (T, N + 2)
33wash, n_tr = 100, 1200
34F, Y = feats[wash:wash + n_tr], u[wash + h:wash + n_tr + h]
35W_out = torch.linalg.solve(F.T @ F + beta * torch.eye(N + 2), F.T @ Y) # ridge, closed form
36
37t0 = wash + n_tr
38pred, target = feats[t0:t0 + 400] @ W_out, u[t0 + h:t0 + h + 400]
39print("test NRMSE", ((pred - target).pow(2).mean().sqrt() / u.std()).item())
40# Library alternative: reservoirpy (Reservoir(units, lr=leak, sr=rho) >> Ridge(ridge=beta))
An ESN in PyTorch: random reservoir, spectral-radius scaling, ridge readout (no autograd)

Use when

  • +Training is one linear solve: seconds, no gradients, no vanishing or exploding gradients
  • +Convex readout with a unique optimum; easy to retrain or adapt online
  • +Strong on chaotic and dynamical time series with little data
  • +Reservoir can be any physical dynamical system

Avoid when

  • −Features are not learned, so it needs large NN and still underperforms trained RNNs on complex tasks
  • −Sensitive to ρ\rho, input scaling and leak rate; tuned by search
  • −Random reservoir: results vary with the seed
  • −Memory capacity is bounded by N; hard to capture very long dependencies

Check yourself

Which parameters of an echo state network are fitted during training?

Check yourself

You raise the spectral radius from 0.9 to 2.0 and the validation error rises sharply. Two copies of the reservoir started from different states no longer converge under the same input. What has been lost?

Remember

  • ✓Random fixed reservoir x(n)=(1−a)x(n−1)+atanh⁡(Win[1;u]+Wx(n−1))x(n)=(1-a)x(n-1)+a\tanh(W_{in}[1;u]+Wx(n-1)); only WoutW_{out} is trained.
  • ✓Readout by ridge regression: Wout=YX⊤(XX⊤+βI)−1W_{out}=YX^\top(XX^\top+\beta I)^{-1}, after a washout.
  • ✓Echo state property: the state must forget its initial condition; ρ(W)<1\rho(W)<1 is the rule of thumb, σmax⁡<1\sigma_{\max}<1 the sufficient condition.
  • ✓ρ\rho sets memory length: too small forgets the input, too large creates self-generated dynamics; the optimum is near the edge of stability.
  • ✓Jaeger (2001) and Maass et al. (2002, liquid state machines) founded reservoir computing; Jaeger and Haas (2004) showed its power on Mackey–Glass.
← back to Deep Learning roadmap