Mamba

A state space model whose step size Δ and matrices B, C depend on the current token, so it can choose what to store and what to ignore. That breaks the convolution trick, so Mamba runs a hardware-aware parallel scan: linear time in training, constant memory per generated token.

15 min read1260 wordsadvanced0 viewsupdated 2026-10-12

Definition

Mamba (Gu and Dao, "Mamba: Linear-Time Sequence Modeling with Selective State Spaces", 2023) is a sequence model built from a selective state space model: a diagonal SSM whose discretisation step Δt\Delta_t and input and output maps BtB_t, CtC_t are computed from the token xtx_t. The selective SSM sits inside a simple gated block (projection, short causal convolution, SiLU, SSM, multiplicative gate) that is stacked homogeneously, with no attention and no separate MLP.

Linear time-invariant SSMs such as S4 (see State space models (SSMs)) can be trained as convolutions and run as RNNs, and they dominate long-range benchmarks. On language they lagged behind Transformers. Gu and Dao located the weakness in content-based reasoning: an LTI model applies the same dynamics to every token, so it cannot decide to skip a filler word or to reset at a document boundary. Two synthetic tasks make this concrete. In selective copying, a few tokens to be remembered are scattered at random positions among noise tokens; a fixed kernel can solve ordinary copying (constant spacing) but not this. Induction heads need associative recall: after seeing "Harry Potter", predict "Potter" the next time "Harry" appears.

The fix is to let the parameters that govern interaction along the sequence depend on the input. That idea is as old as gated RNNs (see Long short-term memory (LSTM) and Gated recurrent unit (GRU)), and Mamba's Theorem 1 shows that a selective SSM with one state is a gated RNN. What was new was making it fast: an input-dependent SSM has no convolution form, so the paper also designs a fused GPU selective scan. The work appeared on arXiv in December 2023; Mamba-2 (Dao and Gu, ICML 2024) followed.

The selective SSM (S6)

Δt=softplus⁡ ⁣(θΔ+sΔ(xt)),Bt=LinearN(xt),Ct=LinearN(xt),Aˉt=eΔtA,Bˉt=(ΔtA)−1(eΔtA−I)ΔtBt\Delta_t=\operatorname{softplus}\!\big(\theta_\Delta+s_\Delta(x_t)\big),\quad B_t=\mathrm{Linear}_N(x_t),\quad C_t=\mathrm{Linear}_N(x_t),\qquad \bar A_t=e^{\Delta_tA},\quad \bar B_t=(\Delta_tA)^{-1}\big(e^{\Delta_tA}-I\big)\Delta_tB_t
xt∈RDx_t\in\mathbb R^{D}
the token's features (inside the block, after expansion to EDED channels)
AA
diagonal, real and not input-dependent: An=−exp⁡(⋅)A_n=-\exp(\cdot), initialised S4D-Real An=−(n+1)A_n=-(n+1); it acts only through eΔtAe^{\Delta_tA}
Δt\Delta_t
per channel and per token; sΔs_\Delta is a low-rank projection, the bias θΔ\theta_\Delta is initialised so that Δ∈[0.001,0.1]\Delta\in[0.001,0.1]
Bt,  Ct∈RNB_t,\;C_t\in\mathbb R^{N}
shared by all channels, computed from the token: what to write into the state and what to read out
NN
state size per channel, 16 by default
ht=Aˉt ht−1+Bˉt xt,yt=Ct ht+D xt,∂ht∂xs=(∏r=s+1tAˉr)Bˉsh_t=\bar A_t\,h_{t-1}+\bar B_t\,x_t,\qquad y_t=C_t\,h_t+D\,x_t,\qquad \frac{\partial h_t}{\partial x_s}=\Big(\prod_{r=s+1}^{t}\bar A_r\Big)\bar B_s
ht∈RED×Nh_t\in\mathbb R^{ED\times N}
the recurrent state: NN numbers per channel, a fixed size however long the context
∂ht/∂xs\partial h_t/\partial x_s
the weight of an old input in the current state: a product of token-dependent decays, not a function of the lag t−st-s alone. This is the lost convolution kernel

In Algorithm 2 of the paper the shapes tell the story: in S4, Aˉ,Bˉ\bar A,\bar B have shape (D, N); in the selective SSM they have shape (B, L, D, N), one copy per batch element and position. The model has moved from time-invariant to time-varying.

Why Δ is a gate

Take one channel with N=1N=1, A=−1A=-1, B=1B=1 and Δt=softplus⁡(zt)\Delta_t=\operatorname{softplus}(z_t) with zt=Linear(xt)z_t=\mathrm{Linear}(x_t). Because e−softplus⁡(z)=1/(1+ez)=1−σ(z)e^{-\operatorname{softplus}(z)}=1/(1+e^{z})=1-\sigma(z):

Aˉt=e−Δt=1−σ(zt)=1−gt,Bˉt=1−e−Δt=gt,ht=(1−gt) ht−1+gt xt\bar A_t=e^{-\Delta_t}=1-\sigma(z_t)=1-g_t,\qquad \bar B_t=1-e^{-\Delta_t}=g_t,\qquad h_t=(1-g_t)\,h_{t-1}+g_t\,x_t
gt=σ(zt)g_t=\sigma(z_t)
a sigmoid gate computed from the token (Mamba's Theorem 1)
Δt→∞\Delta_t\to\infty
gt→1g_t\to1: reset the state and store the current token
Δt→0\Delta_t\to0
gt→0g_t\to0: ignore the token and keep the state unchanged

So a large Δ\Delta means "focus on this input", a small Δ\Delta means "this input is transient, skip it". The paper projects xtx_t to a single number before broadcasting it to Δ\Delta for exactly this reason: if a token should be ignored, all channels should ignore it. Making BtB_t and CtC_t selective adds finer control over what enters the state and what leaves it. This is also why the softplus appears: it keeps Δ\Delta positive and turns the ZOH discretisation into a sigmoid gate.

tokens: ■ marked (keep) ▫ filler (ignore) · click a token to probe h_t−101Δ_tlog scale 0.01…10 · selective Δ_t = softplus(±s) · ┄ LTI Δweight of each input in h_47 = B̄ₛ·Π Āᵣ (bars normalised per row)selLTIMSE selective0.0050MSE LTI (this Δ)0.3053best LTI Δ0.355 → 0.3049gate g on marked / filler0.982 / 0.018h_47: sel / LTI / target-0.41 / 0.21 / -0.50parallel scan: 6 rounds|Δh| 2e-16━ selective h_t ━ LTI h_t ┄ target (latest marked value) · dots = inputs
seed 1
fig.One channel of a selective SSM (N = 1, A = −1, B = 1, the Theorem 1 setting) on a 48-token sequence. The task: hold the value of the latest marked (blue) token and ignore filler. The selective model sets Δₜ = softplus(±s) from the marked flag; the LTI model uses one Δ for every token. At s = 4 the gate is 0.982 on marked tokens and 0.018 on filler, the state follows the dashed target, and the MSE is 0.0050; the LTI model at Δ = 0.3 has MSE 0.3053, and the best fixed Δ (0.355) only reaches 0.3049. The bottom rows show the weight of each input in h₄₇: the selective state puts 0.897 on token 42 (the last marked one) and 0.090 on all filler together, while the LTI weights decay with lag only, 0.259 on the current filler token and 0.941 on filler overall. Set s = 0 and the selective model becomes LTI (gate 0.5, MSE 0.3262). The parallel scan reproduces the sequential result in 6 rounds.

Computing it: the hardware-aware selective scan

Without a fixed kernel there is no FFT. The recurrence ht=Aˉtht−1+Bˉtxth_t=\bar A_th_{t-1}+\bar B_tx_t is still linear in hh, and linear recurrences compose associatively, so they can be evaluated by a parallel prefix scan (Blelloch, 1990) in O(log⁡L)O(\log L) sequential rounds:

(a1,b1)∘(a2,b2)=(a1a2,  a2b1+b2),ht=second component of (Aˉ1,Bˉ1x1)∘⋯∘(Aˉt,Bˉtxt)(a_1,b_1)\circ(a_2,b_2)=(a_1a_2,\;a_2b_1+b_2),\qquad h_t=\text{second component of }(\bar A_1,\bar B_1x_1)\circ\cdots\circ(\bar A_t,\bar B_tx_t)
(a,b)(a,b)
the affine map h↦ah+bh\mapsto ah+b; composing two of them is again affine, which is all a scan needs
∘\circ
associative but not commutative: order along the sequence matters

The real bottleneck is memory traffic. The expanded state has shape (B, L, D, N), NN times larger than the input. Mamba's kernel never writes it to GPU main memory (HBM): it loads Δ\Delta, AA, BB, CC into on-chip SRAM, discretises and scans there (kernel fusion), and writes back only the output of shape (B, L, D). For the backward pass the states are recomputed rather than stored, so the layer's memory matches an optimised Transformer with FlashAttention. The paper reports the scan faster than FlashAttention-2 beyond sequence length 2K and 20–40× faster than a standard scan in PyTorch.

At a glance

Training
O(BLDN)O(BLDN) work, parallel scan over LL; no L2L^2 term
Generation
O(DN)O(DN) per token with a constant-size state: no KV cache that grows with context
Throughput
5× the generation throughput of a Transformer of similar size (paper, Section 4.5)
Defaults
state N=16N=16, expansion E=2E=2, conv width 4, Δ\Delta rank ⌈D/16⌉\lceil D/16\rceil, real diagonal AA
Block params
≈3ED2\approx3ED^2: 2ED22ED^2 in, ED2ED^2 out; two Mamba blocks ≈ one Transformer layer (12D212D^2)
Lineage
S4 → H3 → S6 (selective S4) → Mamba → Mamba-2 (SSD)

The Mamba block

Gu and Dao merge the H3 block (the basis of earlier SSM language models) and a gated MLP into one block. The input of width DD is projected to two branches of width EDED. The main branch passes through a depthwise causal conv1d of width 4, a SiLU, and the selective SSM; the other branch is a SiLU gate. Their product is projected back to DD. Blocks are stacked with residual connections and normalisation (LayerNorm or RMSNorm), and nothing else: no attention and no separate MLP.

partshapeparameters
in_proj (x and z branches)768 → 2 × 15362,359,296
conv1d (depthwise, width 4, bias)1536 × 4 + 15367,680
x_proj (Δ rank 48, B, C)1536 → 48 + 16 + 16122,880
dt_proj (with bias)48 → 153675,264
A_log, D1536 × 16, 153626,112
out_proj1536 → 7681,179,648
total3,770,880
Parameters of one block with D = 768, E = 2, N = 16 (computed by the figure's library, same layout as mamba_ssm.Mamba). 24 such blocks plus a 50,280 × 768 tied embedding give about 129 M parameters, the Mamba-130M model

Worked example: five tokens through a selective gate

One channel, N=1N=1, A=−1A=-1, B=1B=1, values x=(0.9,  1.0,  −0.7,  0.4,  −0.5)x=(0.9,\;1.0,\;-0.7,\;0.4,\;-0.5) where tokens 2 and 5 are marked. The selective model uses zt=+4z_t=+4 on marked tokens and −4-4 on filler; the LTI model uses Δ=ln⁡2\Delta=\ln2 for all, i.e. a constant gate 0.50.5.

Selective scan by hand (checked in Node with the figure's library)

  1. 1Steps: Δ=softplus⁡(±4)=4.0181\Delta=\operatorname{softplus}(\pm4)=4.0181 on marked and 0.01810.0181 on filler tokens. Gates g=1−e−Δ=σ(±4)=0.9820g=1-e^{-\Delta}=\sigma(\pm4)=0.9820 and 0.01800.0180.
  2. 2h1=0.018⋅0.9=0.0162h_1=0.018\cdot0.9=0.0162 (filler barely enters). h2=0.982⋅1.0+0.018⋅0.0162=0.9823h_2=0.982\cdot1.0+0.018\cdot0.0162=0.9823 (marked: almost a full overwrite).
  3. 3h3=0.982⋅0.9823+0.018⋅(−0.7)=0.9520h_3=0.982\cdot0.9823+0.018\cdot(-0.7)=0.9520 and h4=0.9421h_4=0.9421: the stored 1.0 survives two filler tokens. h5=−0.4741h_5=-0.4741 after the next marked token −0.5-0.5.
  4. 4LTI with constant gate 0.5: h=(0.45,  0.725,  0.0125,  0.2063,  −0.1469)h=(0.45,\;0.725,\;0.0125,\;0.2063,\;-0.1469). After token 3 it has already lost the 1.0, because it cannot tell filler from content.

Results

Selective copying (Table 1): S4 with no gating reaches 18.3% accuracy; replacing S4 by the selective S6 lifts the same architecture to 97.0%; the full Mamba block with S6 reaches 99.8%, while the Mamba block with S4 inside gets 56.4%. Selection, not the gated architecture, does the work. Induction heads (Table 2): trained at length 256, Mamba solves the task perfectly and extrapolates to sequences of about one million tokens, 4000× longer, while no other method tested generalises beyond 2× its training length.

modelPile ppl ↓LAMBADA ppl ↓average acc ↑
Pythia-2.8B6.735.0459.1
RWKV-3B7.005.2459.6
Mamba-2.8B6.224.2363.3
Pythia-6.9B6.514.4561.7
Zero-shot evaluation, Gu & Dao (2023), Table 3 (excerpt). Average accuracy over LAMBADA, HellaSwag, PIQA, ARC-E, ARC-C, WinoGrande; Pile is validation perplexity; open-source models trained on up to 300B tokens

The abstract's summary: "our Mamba-3B model outperforms Transformers of the same size and matches Transformers twice its size, both in pretraining and downstream evaluation". The paper also reports strong results on DNA and raw-audio modelling, where performance keeps improving with context up to million-length sequences.

Mamba-2 (Dao and Gu, "Transformers are SSMs", ICML 2024) restricts AA to a scalar times the identity per head and uses a larger head dimension. The resulting state space duality (SSD) shows that this SSM equals a form of masked linear attention, (L∘QK⊤)V(L\circ QK^\top)V with a decay mask Lij=ai⋯aj+1L_{ij}=a_i\cdots a_{j+1}, which can be computed with matrix multiplications. Its core layer is 2–8× faster than Mamba's scan and allows recurrent states 8× the size of Mamba's or more with minimal slowdown.

python
1import math, torch, torch.nn as nn, torch.nn.functional as F
2
3def selective_scan(x, delta, A, B, C, D):
4 """Sequential reference. x, delta: (b, l, d); A: (d, n); B, C: (b, l, n); D: (d,)."""
5 b, l, d = x.shape
6 dA = torch.exp(delta.unsqueeze(-1) * A) # Abar_t = exp(delta_t A): (b, l, d, n)
7 dBx = delta.unsqueeze(-1) * B.unsqueeze(2) * x.unsqueeze(-1) # Bbar_t x_t ~ delta_t B_t x_t (as in mamba_ssm's ref)
8 h = x.new_zeros(b, d, A.shape[1])
9 ys = []
10 for t in range(l): # the fused kernel does this as a parallel scan in SRAM
11 h = dA[:, t] * h + dBx[:, t]
12 ys.append((h * C[:, t].unsqueeze(1)).sum(-1)) # y_t = C_t h_t
13 return torch.stack(ys, dim=1) + x * D
14
15class MambaBlock(nn.Module):
16 def __init__(self, d_model, d_state=16, d_conv=4, expand=2):
17 super().__init__()
18 di, self.n = expand * d_model, d_state
19 self.r = math.ceil(d_model / 16) # rank of the delta projection
20 self.in_proj = nn.Linear(d_model, 2 * di, bias=False)
21 self.conv1d = nn.Conv1d(di, di, d_conv, groups=di, padding=d_conv - 1)
22 self.x_proj = nn.Linear(di, self.r + 2 * d_state, bias=False)
23 self.dt_proj = nn.Linear(self.r, di)
24 dt = torch.exp(torch.rand(di) * (math.log(0.1) - math.log(1e-3)) + math.log(1e-3))
25 with torch.no_grad():
26 self.dt_proj.bias.copy_(dt + torch.log(-torch.expm1(-dt))) # inverse softplus: delta starts in [1e-3, 0.1]
27 self.A_log = nn.Parameter(torch.log(torch.arange(1, d_state + 1, dtype=torch.float32).repeat(di, 1)))
28 self.D = nn.Parameter(torch.ones(di))
29 self.out_proj = nn.Linear(di, d_model, bias=False)
30
31 def forward(self, u): # u: (b, l, d_model)
32 l = u.shape[1]
33 x, z = self.in_proj(u).chunk(2, dim=-1)
34 x = F.silu(self.conv1d(x.transpose(1, 2))[..., :l].transpose(1, 2)) # causal depthwise conv
35 dt, B, C = self.x_proj(x).split([self.r, self.n, self.n], dim=-1) # input-dependent delta, B, C
36 delta = F.softplus(self.dt_proj(dt))
37 A = -torch.exp(self.A_log) # S4D-Real init: A_n = -(n + 1)
38 y = selective_scan(x, delta, A, B, C, self.D)
39 return self.out_proj(y * F.silu(z)) # SiLU gate branch
40
41blk = MambaBlock(768)
42print(sum(p.numel() for p in blk.parameters())) # 3770880
43print(blk(torch.randn(2, 64, 768)).shape) # torch.Size([2, 64, 768])
44
45# Fused, hardware-aware version (pip install mamba-ssm; needs a CUDA GPU):
46# from mamba_ssm import Mamba
47# m = Mamba(d_model=768, d_state=16, d_conv=4, expand=2).to("cuda")
48# y = m(torch.randn(2, 64, 768, device="cuda"))
A readable Mamba block with a sequential reference scan (the fused CUDA kernel is in the mamba-ssm package)

Use when

  • +Linear-time training and constant-memory O(1)-per-token generation: no growing KV cache
  • +Content-aware: Δ, B, C let the model skip filler, store key tokens and reset at boundaries
  • +Matches Transformers about twice its size at 3B scale on the paper's zero-shot suite; 5× generation throughput
  • +Simple homogeneous block (no attention, no separate MLP)

Avoid when

  • −No convolution form: needs a custom fused scan kernel to be fast (CUDA-specific implementations)
  • −Fixed-size state limits exact retrieval and copying of long spans compared with attention
  • −Sequential dependence remains at inference: tokens of one sequence are still processed one at a time
  • −Younger ecosystem; best results often come from hybrids with attention layers

Check yourself

In Mamba's Theorem 1 setting (N = 1, A = −1, B = 1, Δ = softplus(z)), what happens when a token produces a very large Δ?

Check yourself

Why does Mamba need a parallel scan instead of the FFT convolution used to train S4?

Remember

  • ✓Selective SSM: Δt=softplus⁡(θ+sΔ(xt))\Delta_t=\operatorname{softplus}(\theta+s_\Delta(x_t)), Bt,Ct=Linear(xt)B_t,C_t=\mathrm{Linear}(x_t), ht=eΔtAht−1+Bˉtxth_t=e^{\Delta_tA}h_{t-1}+\bar B_tx_t, yt=Ctht+Dxty_t=C_th_t+Dx_t.
  • ✓With one state it is a gated RNN, ht=(1−gt)ht−1+gtxth_t=(1-g_t)h_{t-1}+g_tx_t, gt=σ(zt)g_t=\sigma(z_t): large Δ\Delta stores, small Δ\Delta ignores.
  • ✓Input dependence removes the convolution view; a fused, SRAM-resident parallel scan with recomputation keeps training linear-time and memory-efficient.
  • ✓Mamba block: in_proj to 2ED2ED, causal conv1d + SiLU + S6 on one branch, SiLU gate on the other, out_proj; ≈3ED2\approx3ED^2 params, E=2E=2, N=16N=16.
  • ✓Mamba-2.8B averages 63.3 zero-shot vs 59.1 for Pythia-2.8B and 61.7 for Pythia-6.9B (Table 3); Mamba-2 (SSD) recasts the layer as masked linear attention and is 2–8× faster.
← back to Deep Learning roadmap