Definition
Echo state network (ESN; Jaeger, 2001) is a recurrent network whose recurrent and input weights are drawn at random and never trained. The recurrent layer, called the reservoir, is driven by the input and acts as a nonlinear fading memory of it; the output is a linear readout fitted by ridge regression. Training an RNN is reduced to one linear least-squares problem.
In 2001, training RNNs by backpropagation through time was slow and fragile: gradients vanished or exploded (Bengio et al., 1994), and bifurcations in the dynamics made the loss jump. Herbert Jaeger's GMD report "The 'echo state' approach to analysing and training recurrent neural networks" sidestepped the problem: if the random dynamics already contain the right features of the input history, there is nothing to backpropagate through. Independently, Maass, Natschläger and Markram (2002) proposed the liquid state machine, the same idea with spiking neurons as a model of cortical microcircuits. The two are now grouped as reservoir computing.
The approach became famous with Jaeger and Haas (Science, 2004), who used a 1000-unit ESN on the chaotic Mackey–Glass benchmark and reported that "accuracy is improved by a factor of 2400 over previous techniques", and improved the signal error rate of a nonlinear channel equaliser by two orders of magnitude. Reservoirs remain popular for chaotic time series, for very cheap online learning, and as physical computers (photonic, spintronic, mechanical) where the "reservoir" is a physical system that cannot be trained at all.
The model
- reservoir state; is typically –
- sparse random recurrent matrix, rescaled to a chosen spectral radius ; fixed
- random input weights (and bias), uniform in ; the input scaling sets how nonlinear the units are; fixed
- leak rate (Jaeger et al., 2007, leaky-integrator ESN). is the plain ESN; small slows the dynamics to match slow inputs
- the only trained parameters:
Training collects the states of one long run into a matrix (one column per step, after discarding a washout while the arbitrary initial state fades) and the targets into , then solves the regularised normal equations. Because nothing upstream of changes, this is a convex problem with a unique solution, and its cost is one pass of the reservoir plus a -sized linear solve.
- matrix of collected features
- ridge (Tikhonov) strength; keeps small so the readout does not amplify tiny state differences
The echo state property and the spectral radius
An ESN only works if its state is a function of the input history alone: two copies of the reservoir started from different states and fed the same input must converge. Jaeger called this the echo state property (ESP); the state is an "echo" of the input. Without it the readout would be fitting the accident of the initial condition, and with too much of it the reservoir would forget the input before the readout can use it.
- spectral radius: the asymptotic per-step growth of the linearised dynamics around
- largest singular value; makes the tanh update a contraction (Jaeger's sufficient condition, very conservative)
- eigenvalues of the leaky update's linear part
is neither necessary nor sufficient in general: tanh saturation driven by a strong input can restore contraction when , and non-normal matrices can amplify transients even when . It is still the knob everyone tunes, because it sets the memory time scale: the closer the effective radius is to 1, the slower the echo of an input fades. Lukoševičius (2012) recommends treating , the input scaling and the leak rate as the three main hyperparameters, with near 1 for tasks that need long memory.
At a glance
- Trained
- only , in closed form (ridge regression); no BPTT, no gradients
- Fixed
- (sparse, rescaled to spectral radius ), , bias
- Key hyperparameters
- , input scaling, leak rate , ridge , reservoir size
- Condition
- echo state property: the initial state is forgotten; as rule of thumb
- Cost
- one reservoir run, , plus one -dim linear solve
- Good at
- chaotic and dynamical time series, fast online adaptation, physical/neuromorphic computing
- Relatives
- liquid state machine (Maass et al., 2002); Liquid neural network and Neural ODE are trained continuous-time cousins
Reading the figure: memory versus stability
The task needs memory: to predict 50 steps ahead the readout must see the recent shape of the curve, not just its current value, which is why the input-only predictor fails (NRMSE 0.725). A small makes every echo vanish within a few steps, so the state carries little more than . Raising lengthens the memory and the error drops by more than a factor of 7 up to around with , whose effective radius bound is . Beyond that, the reservoir starts to generate its own dynamics and loses the echo state property: at two initial states never merge, and the readout is fitting noise of its own making.
| setting | test NRMSE | train NRMSE | ‖W_out‖ |
|---|---|---|---|
| , | 0.3122 | 0.3377 | 6328.6 |
| , | 0.1438 | 0.1263 | 2016.2 |
| , (default) | 0.0624 | 0.0528 | 949.6 |
| , | 0.0425 | 0.0346 | 395.1 |
| , | 0.2046 | 0.1657 | 18.0 |
| , (no leak) | 0.1141 | 0.1110 | 2100.0 |
| , , | 0.4719 | 0.4441 | 8.1 |
The leak rate works on time scale: Mackey–Glass varies slowly relative to one sample, and makes each unit a low-pass filter, which beat the plain ESN () by almost a factor of 2 at the same . Ridge matters for a different reason: on this noise-free series a tiny is best for one-step evaluation (1e−10 gives 0.0463), but the readout norm grows to 7727, and a readout that large is fragile when the ESN is run in closed loop as a generator, feeding its prediction back as the next input. Tune on the task you will actually run.
Worked example: an echo fading in a 2-unit reservoir (checked in Node)
- 1Take (eigenvalues , so ), , no bias, , , and one input pulse .
- 2. Then , , , .
- 3The norms shrink by per step, approaching as the state leaves tanh's curved region. The pulse is remembered, rotating between the units, and fades geometrically: an echo.
- 4With leak the effective eigenvalues are , of modulus , and the simulated norms shrink by – per step. The bound is loose here because the eigenvalues are imaginary.
1import torch2torch.set_default_dtype(torch.float64) # ridge with tiny beta needs float643g = torch.Generator().manual_seed(0)45def mackey_glass(n, tau=17, dt=0.1, sub=10, discard=500):6 lag, x, hist, out = int(tau / dt), 1.2, [1.2] * int(tau / dt), []7 for s in range((n + discard) * sub):8 xl = hist[s % lag] # x(t - tau)9 hist[s % lag] = x10 x = x + dt * (0.2 * xl / (1 + xl ** 10) - 0.1 * x)11 if (s + 1) % sub == 0 and (s + 1) // sub > discard:12 out.append(x)13 return torch.tensor(out)1415N, rho, leak, beta, h = 100, 0.9, 0.3, 1e-6, 5016W = torch.randn(N, N, generator=g) * (torch.rand(N, N, generator=g) < 0.1)17W *= rho / torch.linalg.eigvals(W).abs().max() # set the spectral radius18W_in = (torch.rand(N, 2, generator=g) * 2 - 1) * 0.5 # [bias, input] columns1920@torch.no_grad()21def run(u): # u: (T,) -> states (T, N)22 x, xs = torch.zeros(N), []23 for ut in u:24 x_tilde = torch.tanh(W_in @ torch.stack([torch.tensor(1.0), ut]) + W @ x)25 x = (1 - leak) * x + leak * x_tilde26 xs.append(x)27 return torch.stack(xs)2829u = mackey_glass(1900)30u = (u - u.mean()) * 431X = run(u)32feats = torch.cat([torch.ones(len(u), 1), u[:, None], X], dim=1) # (T, N + 2)33wash, n_tr = 100, 120034F, Y = feats[wash:wash + n_tr], u[wash + h:wash + n_tr + h]35W_out = torch.linalg.solve(F.T @ F + beta * torch.eye(N + 2), F.T @ Y) # ridge, closed form3637t0 = wash + n_tr38pred, target = feats[t0:t0 + 400] @ W_out, u[t0 + h:t0 + h + 400]39print("test NRMSE", ((pred - target).pow(2).mean().sqrt() / u.std()).item())40# Library alternative: reservoirpy (Reservoir(units, lr=leak, sr=rho) >> Ridge(ridge=beta))Use when
- +Training is one linear solve: seconds, no gradients, no vanishing or exploding gradients
- +Convex readout with a unique optimum; easy to retrain or adapt online
- +Strong on chaotic and dynamical time series with little data
- +Reservoir can be any physical dynamical system
Avoid when
- −Features are not learned, so it needs large and still underperforms trained RNNs on complex tasks
- −Sensitive to , input scaling and leak rate; tuned by search
- −Random reservoir: results vary with the seed
- −Memory capacity is bounded by N; hard to capture very long dependencies
Check yourself
Which parameters of an echo state network are fitted during training?
Check yourself
You raise the spectral radius from 0.9 to 2.0 and the validation error rises sharply. Two copies of the reservoir started from different states no longer converge under the same input. What has been lost?
Remember
- ✓Random fixed reservoir ; only is trained.
- ✓Readout by ridge regression: , after a washout.
- ✓Echo state property: the state must forget its initial condition; is the rule of thumb, the sufficient condition.
- ✓ sets memory length: too small forgets the input, too large creates self-generated dynamics; the optimum is near the edge of stability.
- ✓Jaeger (2001) and Maass et al. (2002, liquid state machines) founded reservoir computing; Jaeger and Haas (2004) showed its power on Mackey–Glass.