# Mamba

> A state space model whose step size Δ and matrices B, C depend on the current token, so it can choose what to store and what to ignore. That breaks the convolution trick, so Mamba runs a hardware-aware parallel scan: linear time in training, constant memory per generated token.

Source: https://www.mlfoundry.in/learn/mamba  
Section: Deep Learning › Recurrent Neural Networks and Sequence Models  
Level: advanced · 15 min read · updated 2026-10-12

**Mamba:** (Gu and Dao, "Mamba: Linear-Time Sequence Modeling with Selective State Spaces", 2023) is a sequence model built from a **selective state space model**: a diagonal SSM whose discretisation step $\Delta_t$ and input and output maps $B_t$, $C_t$ are computed from the token $x_t$. The selective SSM sits inside a simple gated block (projection, short causal convolution, SiLU, SSM, multiplicative gate) that is stacked homogeneously, with no attention and no separate MLP.

Linear time-invariant SSMs such as S4 (see **State space models (SSMs)**) can be trained as convolutions and run as RNNs, and they dominate long-range benchmarks. On language they lagged behind Transformers. Gu and Dao located the weakness in **content-based reasoning**: an LTI model applies the same dynamics to every token, so it cannot decide to skip a filler word or to reset at a document boundary. Two synthetic tasks make this concrete. In **selective copying**, a few tokens to be remembered are scattered at random positions among noise tokens; a fixed kernel can solve ordinary copying (constant spacing) but not this. **Induction heads** need associative recall: after seeing "Harry Potter", predict "Potter" the next time "Harry" appears.

The fix is to let the parameters that govern interaction along the sequence depend on the input. That idea is as old as gated RNNs (see **Long short-term memory (LSTM)** and **Gated recurrent unit (GRU)**), and Mamba's Theorem 1 shows that a selective SSM with one state *is* a gated RNN. What was new was making it fast: an input-dependent SSM has no convolution form, so the paper also designs a fused GPU **selective scan**. The work appeared on arXiv in December 2023; **Mamba-2** (Dao and Gu, ICML 2024) followed.

## The selective SSM (S6)

$$
\Delta_t=\operatorname{softplus}\!\big(\theta_\Delta+s_\Delta(x_t)\big),\quad B_t=\mathrm{Linear}_N(x_t),\quad C_t=\mathrm{Linear}_N(x_t),\qquad \bar A_t=e^{\Delta_tA},\quad \bar B_t=(\Delta_tA)^{-1}\big(e^{\Delta_tA}-I\big)\Delta_tB_t
$$

- $x_t\in\mathbb R^{D}$: the token's features (inside the block, after expansion to $ED$ channels)
- $A$: diagonal, real and **not** input-dependent: $A_n=-\exp(\cdot)$, initialised S4D-Real $A_n=-(n+1)$; it acts only through $e^{\Delta_tA}$
- $\Delta_t$: per channel and per token; $s_\Delta$ is a low-rank projection, the bias $\theta_\Delta$ is initialised so that $\Delta\in[0.001,0.1]$
- $B_t,\;C_t\in\mathbb R^{N}$: shared by all channels, computed from the token: what to write into the state and what to read out
- $N$: state size per channel, 16 by default

$$
h_t=\bar A_t\,h_{t-1}+\bar B_t\,x_t,\qquad y_t=C_t\,h_t+D\,x_t,\qquad \frac{\partial h_t}{\partial x_s}=\Big(\prod_{r=s+1}^{t}\bar A_r\Big)\bar B_s
$$

- $h_t\in\mathbb R^{ED\times N}$: the recurrent state: $N$ numbers per channel, a fixed size however long the context
- $\partial h_t/\partial x_s$: the weight of an old input in the current state: a product of **token-dependent** decays, not a function of the lag $t-s$ alone. This is the lost convolution kernel

In Algorithm 2 of the paper the shapes tell the story: in S4, $\bar A,\bar B$ have shape (D, N); in the selective SSM they have shape (B, L, D, N), one copy per batch element and position. The model has moved from time-invariant to time-varying.

## Why Δ is a gate

Take one channel with $N=1$, $A=-1$, $B=1$ and $\Delta_t=\operatorname{softplus}(z_t)$ with $z_t=\mathrm{Linear}(x_t)$. Because $e^{-\operatorname{softplus}(z)}=1/(1+e^{z})=1-\sigma(z)$:

$$
\bar A_t=e^{-\Delta_t}=1-\sigma(z_t)=1-g_t,\qquad \bar B_t=1-e^{-\Delta_t}=g_t,\qquad h_t=(1-g_t)\,h_{t-1}+g_t\,x_t
$$

- $g_t=\sigma(z_t)$: a sigmoid gate computed from the token (Mamba's Theorem 1)
- $\Delta_t\to\infty$: $g_t\to1$: **reset** the state and store the current token
- $\Delta_t\to0$: $g_t\to0$: **ignore** the token and keep the state unchanged

So a large $\Delta$ means "focus on this input", a small $\Delta$ means "this input is transient, skip it". The paper projects $x_t$ to a single number before broadcasting it to $\Delta$ for exactly this reason: if a token should be ignored, all channels should ignore it. Making $B_t$ and $C_t$ selective adds finer control over what enters the state and what leaves it. This is also why the **softplus** appears: it keeps $\Delta$ positive and turns the ZOH discretisation into a sigmoid gate.

> Figure: One channel of a selective SSM (N = 1, A = −1, B = 1, the Theorem 1 setting) on a 48-token sequence. The task: hold the value of the latest marked (blue) token and ignore filler. The selective model sets Δₜ = softplus(±s) from the marked flag; the LTI model uses one Δ for every token. At s = 4 the gate is 0.982 on marked tokens and 0.018 on filler, the state follows the dashed target, and the MSE is 0.0050; the LTI model at Δ = 0.3 has MSE 0.3053, and the best fixed Δ (0.355) only reaches 0.3049. The bottom rows show the weight of each input in h₄₇: the selective state puts 0.897 on token 42 (the last marked one) and 0.090 on all filler together, while the LTI weights decay with lag only, 0.259 on the current filler token and 0.941 on filler overall. Set s = 0 and the selective model becomes LTI (gate 0.5, MSE 0.3262). The parallel scan reproduces the sequential result in 6 rounds.

## Computing it: the hardware-aware selective scan

Without a fixed kernel there is no FFT. The recurrence $h_t=\bar A_th_{t-1}+\bar B_tx_t$ is still **linear in $h$**, and linear recurrences compose associatively, so they can be evaluated by a parallel prefix scan (Blelloch, 1990) in $O(\log L)$ sequential rounds:

$$
(a_1,b_1)\circ(a_2,b_2)=(a_1a_2,\;a_2b_1+b_2),\qquad h_t=\text{second component of }(\bar A_1,\bar B_1x_1)\circ\cdots\circ(\bar A_t,\bar B_tx_t)
$$

- $(a,b)$: the affine map $h\mapsto ah+b$; composing two of them is again affine, which is all a scan needs
- $\circ$: associative but not commutative: order along the sequence matters

The real bottleneck is memory traffic. The expanded state has shape (B, L, D, N), $N$ times larger than the input. Mamba's kernel never writes it to GPU main memory (HBM): it loads $\Delta$, $A$, $B$, $C$ into on-chip SRAM, discretises and scans there (**kernel fusion**), and writes back only the output of shape (B, L, D). For the backward pass the states are **recomputed** rather than stored, so the layer's memory matches an optimised Transformer with FlashAttention. The paper reports the scan faster than FlashAttention-2 beyond sequence length 2K and 20–40× faster than a standard scan in PyTorch.

**At a glance**

- **Training:** $O(BLDN)$ work, parallel scan over $L$; no $L^2$ term
- **Generation:** $O(DN)$ per token with a constant-size state: no KV cache that grows with context
- **Throughput:** 5× the generation throughput of a Transformer of similar size (paper, Section 4.5)
- **Defaults:** state $N=16$, expansion $E=2$, conv width 4, $\Delta$ rank $\lceil D/16\rceil$, real diagonal $A$
- **Block params:** $\approx3ED^2$: $2ED^2$ in, $ED^2$ out; two Mamba blocks ≈ one Transformer layer ($12D^2$)
- **Lineage:** S4 → H3 → S6 (selective S4) → Mamba → Mamba-2 (SSD)

## The Mamba block

Gu and Dao merge the H3 block (the basis of earlier SSM language models) and a gated MLP into one block. The input of width $D$ is projected to two branches of width $ED$. The main branch passes through a depthwise causal **conv1d** of width 4, a **SiLU**, and the selective SSM; the other branch is a SiLU gate. Their product is projected back to $D$. Blocks are stacked with residual connections and normalisation (LayerNorm or RMSNorm), and nothing else: no attention and no separate MLP.

Parameters of one block with D = 768, E = 2, N = 16 (computed by the figure's library, same layout as mamba_ssm.Mamba). 24 such blocks plus a 50,280 × 768 tied embedding give about 129 M parameters, the Mamba-130M model

| part | shape | parameters |
| --- | --- | --- |
| in_proj (x and z branches) | 768 → 2 × 1536 | 2,359,296 |
| conv1d (depthwise, width 4, bias) | 1536 × 4 + 1536 | 7,680 |
| x_proj (Δ rank 48, B, C) | 1536 → 48 + 16 + 16 | 122,880 |
| dt_proj (with bias) | 48 → 1536 | 75,264 |
| A_log, D | 1536 × 16, 1536 | 26,112 |
| out_proj | 1536 → 768 | 1,179,648 |
| total |  | 3,770,880 |

## Worked example: five tokens through a selective gate

One channel, $N=1$, $A=-1$, $B=1$, values $x=(0.9,\;1.0,\;-0.7,\;0.4,\;-0.5)$ where tokens 2 and 5 are marked. The selective model uses $z_t=+4$ on marked tokens and $-4$ on filler; the LTI model uses $\Delta=\ln2$ for all, i.e. a constant gate $0.5$.

**Selective scan by hand (checked in Node with the figure's library)**

1. Steps: $\Delta=\operatorname{softplus}(\pm4)=4.0181$ on marked and $0.0181$ on filler tokens. Gates $g=1-e^{-\Delta}=\sigma(\pm4)=0.9820$ and $0.0180$.
2. $h_1=0.018\cdot0.9=0.0162$ (filler barely enters). $h_2=0.982\cdot1.0+0.018\cdot0.0162=0.9823$ (marked: almost a full overwrite).
3. $h_3=0.982\cdot0.9823+0.018\cdot(-0.7)=0.9520$ and $h_4=0.9421$: the stored 1.0 survives two filler tokens. $h_5=-0.4741$ after the next marked token $-0.5$.
4. LTI with constant gate 0.5: $h=(0.45,\;0.725,\;0.0125,\;0.2063,\;-0.1469)$. After token 3 it has already lost the 1.0, because it cannot tell filler from content.

## Results

**Selective copying** (Table 1): S4 with no gating reaches 18.3% accuracy; replacing S4 by the selective S6 lifts the same architecture to 97.0%; the full Mamba block with S6 reaches 99.8%, while the Mamba block with S4 inside gets 56.4%. Selection, not the gated architecture, does the work. **Induction heads** (Table 2): trained at length 256, Mamba solves the task perfectly and extrapolates to sequences of about one million tokens, 4000× longer, while no other method tested generalises beyond 2× its training length.

Zero-shot evaluation, Gu & Dao (2023), Table 3 (excerpt). Average accuracy over LAMBADA, HellaSwag, PIQA, ARC-E, ARC-C, WinoGrande; Pile is validation perplexity; open-source models trained on up to 300B tokens

| model | Pile ppl ↓ | LAMBADA ppl ↓ | average acc ↑ |
| --- | --- | --- | --- |
| Pythia-2.8B | 6.73 | 5.04 | 59.1 |
| RWKV-3B | 7.00 | 5.24 | 59.6 |
| Mamba-2.8B | 6.22 | 4.23 | 63.3 |
| Pythia-6.9B | 6.51 | 4.45 | 61.7 |

The abstract's summary: "our Mamba-3B model outperforms Transformers of the same size and matches Transformers twice its size, both in pretraining and downstream evaluation". The paper also reports strong results on DNA and raw-audio modelling, where performance keeps improving with context up to million-length sequences.

**Mamba-2** (Dao and Gu, "Transformers are SSMs", ICML 2024) restricts $A$ to a scalar times the identity per head and uses a larger head dimension. The resulting **state space duality (SSD)** shows that this SSM equals a form of masked linear attention, $(L\circ QK^\top)V$ with a decay mask $L_{ij}=a_i\cdots a_{j+1}$, which can be computed with matrix multiplications. Its core layer is 2–8× faster than Mamba's scan and allows recurrent states 8× the size of Mamba's or more with minimal slowdown.

```python
import math, torch, torch.nn as nn, torch.nn.functional as F

def selective_scan(x, delta, A, B, C, D):
    """Sequential reference. x, delta: (b, l, d); A: (d, n); B, C: (b, l, n); D: (d,)."""
    b, l, d = x.shape
    dA = torch.exp(delta.unsqueeze(-1) * A)                         # Abar_t = exp(delta_t A): (b, l, d, n)
    dBx = delta.unsqueeze(-1) * B.unsqueeze(2) * x.unsqueeze(-1)    # Bbar_t x_t ~ delta_t B_t x_t (as in mamba_ssm's ref)
    h = x.new_zeros(b, d, A.shape[1])
    ys = []
    for t in range(l):                                              # the fused kernel does this as a parallel scan in SRAM
        h = dA[:, t] * h + dBx[:, t]
        ys.append((h * C[:, t].unsqueeze(1)).sum(-1))               # y_t = C_t h_t
    return torch.stack(ys, dim=1) + x * D

class MambaBlock(nn.Module):
    def __init__(self, d_model, d_state=16, d_conv=4, expand=2):
        super().__init__()
        di, self.n = expand * d_model, d_state
        self.r = math.ceil(d_model / 16)                            # rank of the delta projection
        self.in_proj = nn.Linear(d_model, 2 * di, bias=False)
        self.conv1d = nn.Conv1d(di, di, d_conv, groups=di, padding=d_conv - 1)
        self.x_proj = nn.Linear(di, self.r + 2 * d_state, bias=False)
        self.dt_proj = nn.Linear(self.r, di)
        dt = torch.exp(torch.rand(di) * (math.log(0.1) - math.log(1e-3)) + math.log(1e-3))
        with torch.no_grad():
            self.dt_proj.bias.copy_(dt + torch.log(-torch.expm1(-dt)))   # inverse softplus: delta starts in [1e-3, 0.1]
        self.A_log = nn.Parameter(torch.log(torch.arange(1, d_state + 1, dtype=torch.float32).repeat(di, 1)))
        self.D = nn.Parameter(torch.ones(di))
        self.out_proj = nn.Linear(di, d_model, bias=False)

    def forward(self, u):                                           # u: (b, l, d_model)
        l = u.shape[1]
        x, z = self.in_proj(u).chunk(2, dim=-1)
        x = F.silu(self.conv1d(x.transpose(1, 2))[..., :l].transpose(1, 2))   # causal depthwise conv
        dt, B, C = self.x_proj(x).split([self.r, self.n, self.n], dim=-1)    # input-dependent delta, B, C
        delta = F.softplus(self.dt_proj(dt))
        A = -torch.exp(self.A_log)                                  # S4D-Real init: A_n = -(n + 1)
        y = selective_scan(x, delta, A, B, C, self.D)
        return self.out_proj(y * F.silu(z))                         # SiLU gate branch

blk = MambaBlock(768)
print(sum(p.numel() for p in blk.parameters()))                     # 3770880
print(blk(torch.randn(2, 64, 768)).shape)                           # torch.Size([2, 64, 768])

# Fused, hardware-aware version (pip install mamba-ssm; needs a CUDA GPU):
# from mamba_ssm import Mamba
# m = Mamba(d_model=768, d_state=16, d_conv=4, expand=2).to("cuda")
# y = m(torch.randn(2, 64, 768, device="cuda"))
```

A readable Mamba block with a sequential reference scan (the fused CUDA kernel is in the mamba-ssm package)

> **Expecting attention-style exact recall:** The state is a fixed $ED\times N$ summary; a selective SSM decides what to keep, but it cannot keep everything. Tasks that need verbatim lookup of arbitrary earlier tokens (long phone-book style retrieval, many-shot copying) remain harder for pure SSMs than for attention, which is why many recent large models are **hybrids** that interleave a few attention layers with Mamba layers.

> **Running the reference scan in production:** The Python loop above materialises the (b, l, d, n) tensors and loops over $l$; it is for understanding and testing only. Use the fused `selective_scan_fn` from `mamba_ssm` (or `Mamba`/`Mamba2` modules) for real training, and its `step` path with an inference cache for generation, where each new token costs one state update.

> **Selection is compression:** The paper frames sequence modelling as compressing context into a state. Attention does not compress at all (the KV cache keeps every token), so it is effective but its inference cost grows with context. An LTI recurrence compresses with fixed rules, so it is efficient but cannot choose. Selection is the middle path: a small state, filled by content-dependent rules.

> **Related articles:** Selective gating is the SSM version of the gates in **Long short-term memory (LSTM)**, **Gated recurrent unit (GRU)** and **Highway network**; the continuous-time view connects to **Neural ODE** and **Liquid neural network**. The softplus that produces $\Delta$ is covered in **Softplus**.

**Strengths**

- Linear-time training and constant-memory O(1)-per-token generation: no growing KV cache
- Content-aware: Δ, B, C let the model skip filler, store key tokens and reset at boundaries
- Matches Transformers about twice its size at 3B scale on the paper's zero-shot suite; 5× generation throughput
- Simple homogeneous block (no attention, no separate MLP)

**Limitations**

- No convolution form: needs a custom fused scan kernel to be fast (CUDA-specific implementations)
- Fixed-size state limits exact retrieval and copying of long spans compared with attention
- Sequential dependence remains at inference: tokens of one sequence are still processed one at a time
- Younger ecosystem; best results often come from hybrids with attention layers

**Question:** In Mamba's Theorem 1 setting (N = 1, A = −1, B = 1, Δ = softplus(z)), what happens when a token produces a very large Δ?

- A. The state is kept and the token is ignored
- B. The gate g = σ(z) approaches 1: the old state is forgotten and the token is written in
- C. The state explodes because Ā > 1
- D. Nothing: Δ only affects the output map C

**Answer:** B. The gate g = σ(z) approaches 1: the old state is forgotten and the token is written in — $\bar A=e^{-\Delta}\to0$ and $\bar B=1-e^{-\Delta}\to1$, so $h_t\to x_t$: a reset. A small $\Delta$ does the opposite ($\bar A\to1$, $\bar B\to0$). Because $\Delta>0$ and $A<0$, $|\bar A|<1$ always, so the state never explodes.

**Question:** Why does Mamba need a parallel scan instead of the FFT convolution used to train S4?

- A. Its state is complex-valued
- B. Its parameters Δ, B, C change from token to token, so there is no single convolution kernel; but the recurrence is still linear in h and therefore associative
- C. FFTs cannot be computed on GPUs
- D. The scan uses less arithmetic than any convolution

**Answer:** B. Its parameters Δ, B, C change from token to token, so there is no single convolution kernel; but the recurrence is still linear in h and therefore associative — The weight of $x_s$ in $h_t$ is $\big(\prod_{r=s+1}^t\bar A_r\big)\bar B_s$, which depends on the tokens in between, not just on $t-s$. The affine maps $h\mapsto\bar A_th+\bar B_tx_t$ compose associatively, so a prefix scan computes all $h_t$ in $O(\log L)$ parallel rounds.

**Key points**

- Selective SSM: $\Delta_t=\operatorname{softplus}(\theta+s_\Delta(x_t))$, $B_t,C_t=\mathrm{Linear}(x_t)$, $h_t=e^{\Delta_tA}h_{t-1}+\bar B_tx_t$, $y_t=C_th_t+Dx_t$.
- With one state it is a gated RNN, $h_t=(1-g_t)h_{t-1}+g_tx_t$, $g_t=\sigma(z_t)$: large $\Delta$ stores, small $\Delta$ ignores.
- Input dependence removes the convolution view; a fused, SRAM-resident parallel scan with recomputation keeps training linear-time and memory-efficient.
- Mamba block: in_proj to $2ED$, causal conv1d + SiLU + S6 on one branch, SiLU gate on the other, out_proj; $\approx3ED^2$ params, $E=2$, $N=16$.
- Mamba-2.8B averages 63.3 zero-shot vs 59.1 for Pythia-2.8B and 61.7 for Pythia-6.9B (Table 3); Mamba-2 (SSD) recasts the layer as masked linear attention and is 2–8× faster.

---

From MLFoundry (https://www.mlfoundry.in). The web page adds interactive figures and diagrams.
