# Echo state network

> A large, fixed, random recurrent reservoir turns the input history into a rich state, and only a linear readout is trained, by ridge regression in closed form. It works when the reservoir forgets its initial state, typically with a spectral radius near 1.

Source: https://www.mlfoundry.in/learn/echo-state-network  
Section: Deep Learning › Recurrent Neural Networks and Sequence Models  
Level: advanced · 12 min read · updated 2026-10-12

**Echo state network:** (ESN; Jaeger, 2001) is a recurrent network whose recurrent and input weights are drawn at random and **never trained**. The recurrent layer, called the **reservoir**, is driven by the input and acts as a nonlinear fading memory of it; the output is a linear readout $y(n)=W_{out}[1;u(n);x(n)]$ fitted by ridge regression. Training an RNN is reduced to one linear least-squares problem.

In 2001, training RNNs by backpropagation through time was slow and fragile: gradients vanished or exploded (Bengio et al., 1994), and bifurcations in the dynamics made the loss jump. Herbert Jaeger's GMD report "The 'echo state' approach to analysing and training recurrent neural networks" sidestepped the problem: if the random dynamics already contain the right features of the input history, there is nothing to backpropagate through. Independently, Maass, Natschläger and Markram (2002) proposed the **liquid state machine**, the same idea with spiking neurons as a model of cortical microcircuits. The two are now grouped as **reservoir computing**.

The approach became famous with Jaeger and Haas (Science, 2004), who used a 1000-unit ESN on the chaotic Mackey–Glass benchmark and reported that "accuracy is improved by a factor of 2400 over previous techniques", and improved the signal error rate of a nonlinear channel equaliser by two orders of magnitude. Reservoirs remain popular for chaotic time series, for very cheap online learning, and as physical computers (photonic, spintronic, mechanical) where the "reservoir" is a physical system that cannot be trained at all.

## The model

$$
\tilde x(n)=\tanh\!\big(W_{in}[1;u(n)]+W\,x(n-1)\big),\qquad x(n)=(1-a)\,x(n-1)+a\,\tilde x(n),\qquad \hat y(n)=W_{out}\,[1;u(n);x(n)]
$$

- $x(n)\in\mathbb R^{N}$: reservoir state; $N$ is typically $10^2$–$10^4$
- $W$: sparse random $N\times N$ recurrent matrix, rescaled to a chosen spectral radius $\rho(W)$; fixed
- $W_{in}$: random input weights (and bias), uniform in $[-s,s]$; the input scaling $s$ sets how nonlinear the units are; fixed
- $a\in(0,1]$: leak rate (Jaeger et al., 2007, leaky-integrator ESN). $a=1$ is the plain ESN; small $a$ slows the dynamics to match slow inputs
- $W_{out}$: the only trained parameters: $D_y\times(1+D_u+N)$

Training collects the states of one long run into a matrix $X$ (one column $[1;u(n);x(n)]$ per step, after discarding a **washout** while the arbitrary initial state fades) and the targets into $Y$, then solves the regularised normal equations. Because nothing upstream of $W_{out}$ changes, this is a convex problem with a unique solution, and its cost is one pass of the reservoir plus a $(1+D_u+N)$-sized linear solve.

$$
W_{out}=Y X^\top\big(XX^\top+\beta I\big)^{-1}
$$

- $X$: $(1+D_u+N)\times T_{train}$ matrix of collected features
- $\beta$: ridge (Tikhonov) strength; keeps $\lVert W_{out}\rVert$ small so the readout does not amplify tiny state differences

## The echo state property and the spectral radius

An ESN only works if its state is a function of the **input history alone**: two copies of the reservoir started from different states and fed the same input must converge. Jaeger called this the **echo state property** (ESP); the state is an "echo" of the input. Without it the readout would be fitting the accident of the initial condition, and with too much of it the reservoir would forget the input before the readout can use it.

$$
\rho(W)=\max_k|\lambda_k(W)|<1\quad(\text{rule of thumb}),\qquad \sigma_{\max}(W)<1\;\Rightarrow\;\text{ESP for every input},\qquad \text{leaky: }|\lambda_k|_{\text{eff}}=|(1-a)+a\lambda_k|\le(1-a)+a\rho
$$

- $\rho(W)$: spectral radius: the asymptotic per-step growth of the linearised dynamics around $x=0$
- $\sigma_{\max}(W)$: largest singular value; $<1$ makes the tanh update a contraction (Jaeger's sufficient condition, very conservative)
- $(1-a)+a\lambda$: eigenvalues of the leaky update's linear part $(1-a)I+aW$

$\rho(W)<1$ is neither necessary nor sufficient in general: tanh saturation driven by a strong input can restore contraction when $\rho>1$, and non-normal matrices can amplify transients even when $\rho<1$. It is still the knob everyone tunes, because it sets the **memory time scale**: the closer the effective radius is to 1, the slower the echo of an input fades. Lukoševičius (2012) recommends treating $\rho$, the input scaling and the leak rate as the three main hyperparameters, with $\rho$ near 1 for tasks that need long memory.

**At a glance**

- **Trained:** only $W_{out}$, in closed form (ridge regression); no BPTT, no gradients
- **Fixed:** $W$ (sparse, rescaled to spectral radius $\rho$), $W_{in}$, bias
- **Key hyperparameters:** $\rho(W)$, input scaling, leak rate $a$, ridge $\beta$, reservoir size $N$
- **Condition:** echo state property: the initial state is forgotten; $\rho<1$ as rule of thumb
- **Cost:** one reservoir run, $O(T\,N\cdot\text{nnz per row})$, plus one $(N+1+D_u)$-dim linear solve
- **Good at:** chaotic and dynamical time series, fast online adaptation, physical/neuromorphic computing
- **Relatives:** liquid state machine (Maass et al., 2002); **Liquid neural network** and **Neural ODE** are trained continuous-time cousins

> Figure: A 100-unit ESN (10% sparse, tanh, leak rate a) driven by the Mackey–Glass series (τ = 17), with a ridge readout trained on 1200 steps to predict the value h steps ahead and tested on the next 400. The random W has spectral radius 3.37 and is rescaled to ρ. At the defaults ρ = 0.9, a = 0.3, β = 1e−6, h = 50, test NRMSE is 0.0624 (train 0.0528), while the best linear predictor from u(n) alone gets 0.725. Slide ρ: 0.1 gives 0.312 (the reservoir forgets too fast), 1.2 gives 0.0425, 2.0 gives 0.205. The lower-right panel runs the reservoir from two random states with the same input: at ρ = 0.9 their distance falls from 6.0 to 3e−8 in 200 steps (echo state property); at ρ = 2.0 it is still 2.3. Without leakage (a = 1) the default ρ gives 0.114.

## Reading the figure: memory versus stability

The task needs memory: to predict 50 steps ahead the readout must see the recent shape of the curve, not just its current value, which is why the input-only predictor fails (NRMSE 0.725). A small $\rho$ makes every echo vanish within a few steps, so the state carries little more than $u(n)$. Raising $\rho$ lengthens the memory and the error drops by more than a factor of 7 up to around $\rho\approx1.2$ with $a=0.3$, whose effective radius bound is $0.7+0.3\cdot1.2=1.06$. Beyond that, the reservoir starts to generate its own dynamics and loses the echo state property: at $\rho=2$ two initial states never merge, and the readout is fitting noise of its own making.

Test NRMSE for the 50-step-ahead task (figure's lib, run in Node; β = 1e−6 unless stated)

| setting | test NRMSE | train NRMSE | ‖W_out‖ |
| --- | --- | --- | --- |
| $\rho=0.1$, $a=0.3$ | 0.3122 | 0.3377 | 6328.6 |
| $\rho=0.5$, $a=0.3$ | 0.1438 | 0.1263 | 2016.2 |
| $\rho=0.9$, $a=0.3$ (default) | 0.0624 | 0.0528 | 949.6 |
| $\rho=1.2$, $a=0.3$ | 0.0425 | 0.0346 | 395.1 |
| $\rho=2.0$, $a=0.3$ | 0.2046 | 0.1657 | 18.0 |
| $\rho=0.9$, $a=1$ (no leak) | 0.1141 | 0.1110 | 2100.0 |
| $\rho=0.9$, $a=0.3$, $\beta=1$ | 0.4719 | 0.4441 | 8.1 |

The leak rate works on time scale: Mackey–Glass varies slowly relative to one sample, and $a=0.3$ makes each unit a low-pass filter, which beat the plain ESN ($a=1$) by almost a factor of 2 at the same $\rho$. Ridge matters for a different reason: on this noise-free series a tiny $\beta$ is best for one-step evaluation (1e−10 gives 0.0463), but the readout norm grows to 7727, and a readout that large is fragile when the ESN is run in closed loop as a generator, feeding its prediction back as the next input. Tune $\beta$ on the task you will actually run.

**Worked example: an echo fading in a 2-unit reservoir (checked in Node)**

1. Take $W=\begin{pmatrix}0&0.8\\-0.8&0\end{pmatrix}$ (eigenvalues $\pm0.8i$, so $\rho=0.8$), $W_{in}=(1,0)^\top$, no bias, $a=1$, $x(0)=0$, and one input pulse $u=(1,0,0,0,0)$.
2. $x(1)=\tanh(1,0)=(0.7616,\,0)$. Then $x(2)=\tanh(Wx(1))=\tanh(0,-0.6093)=(0,\,-0.5436)$, $x(3)=(-0.4094,\,0)$, $x(4)=(0,\,0.3163)$, $x(5)=(0.2478,\,0)$.
3. The norms $0.762, 0.544, 0.409, 0.316, 0.248$ shrink by $0.714, 0.753, 0.773, 0.783$ per step, approaching $\rho=0.8$ as the state leaves tanh's curved region. The pulse is remembered, rotating between the units, and fades geometrically: an echo.
4. With leak $a=0.5$ the effective eigenvalues are $0.5\pm0.4i$, of modulus $0.640$, and the simulated norms $0.381, 0.241, 0.154, 0.098, 0.063$ shrink by $0.63$–$0.64$ per step. The bound $(1-a)+a\rho=0.9$ is loose here because the eigenvalues are imaginary.

```python
import torch
torch.set_default_dtype(torch.float64)          # ridge with tiny beta needs float64
g = torch.Generator().manual_seed(0)

def mackey_glass(n, tau=17, dt=0.1, sub=10, discard=500):
    lag, x, hist, out = int(tau / dt), 1.2, [1.2] * int(tau / dt), []
    for s in range((n + discard) * sub):
        xl = hist[s % lag]                          # x(t - tau)
        hist[s % lag] = x
        x = x + dt * (0.2 * xl / (1 + xl ** 10) - 0.1 * x)
        if (s + 1) % sub == 0 and (s + 1) // sub > discard:
            out.append(x)
    return torch.tensor(out)

N, rho, leak, beta, h = 100, 0.9, 0.3, 1e-6, 50
W = torch.randn(N, N, generator=g) * (torch.rand(N, N, generator=g) < 0.1)
W *= rho / torch.linalg.eigvals(W).abs().max()  # set the spectral radius
W_in = (torch.rand(N, 2, generator=g) * 2 - 1) * 0.5   # [bias, input] columns

@torch.no_grad()
def run(u):                                       # u: (T,) -> states (T, N)
    x, xs = torch.zeros(N), []
    for ut in u:
        x_tilde = torch.tanh(W_in @ torch.stack([torch.tensor(1.0), ut]) + W @ x)
        x = (1 - leak) * x + leak * x_tilde
        xs.append(x)
    return torch.stack(xs)

u = mackey_glass(1900)
u = (u - u.mean()) * 4
X = run(u)
feats = torch.cat([torch.ones(len(u), 1), u[:, None], X], dim=1)        # (T, N + 2)
wash, n_tr = 100, 1200
F, Y = feats[wash:wash + n_tr], u[wash + h:wash + n_tr + h]
W_out = torch.linalg.solve(F.T @ F + beta * torch.eye(N + 2), F.T @ Y)   # ridge, closed form

t0 = wash + n_tr
pred, target = feats[t0:t0 + 400] @ W_out, u[t0 + h:t0 + h + 400]
print("test NRMSE", ((pred - target).pow(2).mean().sqrt() / u.std()).item())
# Library alternative: reservoirpy (Reservoir(units, lr=leak, sr=rho) >> Ridge(ridge=beta))
```

An ESN in PyTorch: random reservoir, spectral-radius scaling, ridge readout (no autograd)

> **Spectral radius is not the whole story:** Rescaling $W$ to $\rho=0.9$ does not by itself guarantee forgetting: a large input scaling or bias pushes units into saturation and changes the effective dynamics, and a non-normal $W$ can amplify transients for many steps. Check the echo state property empirically as the figure does (two initial states, same input, distance should vanish), discard a washout of at least a few memory lengths, and always pick $\rho$, input scaling and $a$ by validation error, not by the rule of thumb alone.

> **Why random features are enough:** The reservoir is a random, nonlinear, fading-memory feature map of the input history, and the readout is a linear model on those features. It is the time-series counterpart of a random-features kernel machine: if $N$ is large and the dynamics are near the edge of stability, the span of the states contains good approximations of many functions of the recent past, and least squares finds the best one. What it cannot do is shape the features to the task, which is exactly what BPTT-trained **Long short-term memory (LSTM)** networks do.

> **Where ESNs stand today:** For language and other large-data tasks, gated RNNs, Transformers and **State space models (SSMs)** trained end to end are far stronger. ESNs remain competitive for small-data dynamical systems (chaotic forecasting, control), for online learning with recursive least squares, and wherever the reservoir is physical hardware. Modern SSMs share their structure: a linear(ised) recurrent state with carefully chosen eigenvalues, but with the recurrence learned.

**Strengths**

- Training is one linear solve: seconds, no gradients, no vanishing or exploding gradients
- Convex readout with a unique optimum; easy to retrain or adapt online
- Strong on chaotic and dynamical time series with little data
- Reservoir can be any physical dynamical system

**Limitations**

- Features are not learned, so it needs large $N$ and still underperforms trained RNNs on complex tasks
- Sensitive to $\rho$, input scaling and leak rate; tuned by search
- Random reservoir: results vary with the seed
- Memory capacity is bounded by N; hard to capture very long dependencies

**Question:** Which parameters of an echo state network are fitted during training?

- A. All weights, by backpropagation through time
- B. Only the recurrent matrix W, by ridge regression
- C. Only the readout W_out, by ridge regression
- D. W_in and W_out, by gradient descent

**Answer:** C. Only the readout W_out, by ridge regression — $W_{in}$ and $W$ are random and fixed. The states are collected from one run and $W_{out}=YX^\top(XX^\top+\beta I)^{-1}$ is solved in closed form. This is what makes ESNs fast and free of gradient problems.

**Question:** You raise the spectral radius from 0.9 to 2.0 and the validation error rises sharply. Two copies of the reservoir started from different states no longer converge under the same input. What has been lost?

- A. The echo state property: the state now depends on the initial condition, not only on the input history
- B. Sparsity of the reservoir
- C. Linearity of the readout
- D. Nothing; the ridge parameter must be lowered

**Answer:** A. The echo state property: the state now depends on the initial condition, not only on the input history — With $\rho$ well above 1 the reservoir generates its own dynamics and never forgets where it started (in the figure, distance 2.3 after 200 steps). The readout can no longer rely on states being a function of the input, so the fit degrades.

**Key points**

- Random fixed reservoir $x(n)=(1-a)x(n-1)+a\tanh(W_{in}[1;u]+Wx(n-1))$; only $W_{out}$ is trained.
- Readout by ridge regression: $W_{out}=YX^\top(XX^\top+\beta I)^{-1}$, after a washout.
- Echo state property: the state must forget its initial condition; $\rho(W)<1$ is the rule of thumb, $\sigma_{\max}<1$ the sufficient condition.
- $\rho$ sets memory length: too small forgets the input, too large creates self-generated dynamics; the optimum is near the edge of stability.
- Jaeger (2001) and Maass et al. (2002, liquid state machines) founded reservoir computing; Jaeger and Haas (2004) showed its power on Mackey–Glass.

---

From MLFoundry (https://www.mlfoundry.in). The web page adds interactive figures and diagrams.
