Definition
Mamba (Gu and Dao, "Mamba: Linear-Time Sequence Modeling with Selective State Spaces", 2023) is a sequence model built from a selective state space model: a diagonal SSM whose discretisation step and input and output maps , are computed from the token . The selective SSM sits inside a simple gated block (projection, short causal convolution, SiLU, SSM, multiplicative gate) that is stacked homogeneously, with no attention and no separate MLP.
Linear time-invariant SSMs such as S4 (see State space models (SSMs)) can be trained as convolutions and run as RNNs, and they dominate long-range benchmarks. On language they lagged behind Transformers. Gu and Dao located the weakness in content-based reasoning: an LTI model applies the same dynamics to every token, so it cannot decide to skip a filler word or to reset at a document boundary. Two synthetic tasks make this concrete. In selective copying, a few tokens to be remembered are scattered at random positions among noise tokens; a fixed kernel can solve ordinary copying (constant spacing) but not this. Induction heads need associative recall: after seeing "Harry Potter", predict "Potter" the next time "Harry" appears.
The fix is to let the parameters that govern interaction along the sequence depend on the input. That idea is as old as gated RNNs (see Long short-term memory (LSTM) and Gated recurrent unit (GRU)), and Mamba's Theorem 1 shows that a selective SSM with one state is a gated RNN. What was new was making it fast: an input-dependent SSM has no convolution form, so the paper also designs a fused GPU selective scan. The work appeared on arXiv in December 2023; Mamba-2 (Dao and Gu, ICML 2024) followed.
The selective SSM (S6)
- the token's features (inside the block, after expansion to channels)
- diagonal, real and not input-dependent: , initialised S4D-Real ; it acts only through
- per channel and per token; is a low-rank projection, the bias is initialised so that
- shared by all channels, computed from the token: what to write into the state and what to read out
- state size per channel, 16 by default
- the recurrent state: numbers per channel, a fixed size however long the context
- the weight of an old input in the current state: a product of token-dependent decays, not a function of the lag alone. This is the lost convolution kernel
In Algorithm 2 of the paper the shapes tell the story: in S4, have shape (D, N); in the selective SSM they have shape (B, L, D, N), one copy per batch element and position. The model has moved from time-invariant to time-varying.
Why Δ is a gate
Take one channel with , , and with . Because :
- a sigmoid gate computed from the token (Mamba's Theorem 1)
- : reset the state and store the current token
- : ignore the token and keep the state unchanged
So a large means "focus on this input", a small means "this input is transient, skip it". The paper projects to a single number before broadcasting it to for exactly this reason: if a token should be ignored, all channels should ignore it. Making and selective adds finer control over what enters the state and what leaves it. This is also why the softplus appears: it keeps positive and turns the ZOH discretisation into a sigmoid gate.
Computing it: the hardware-aware selective scan
Without a fixed kernel there is no FFT. The recurrence is still linear in , and linear recurrences compose associatively, so they can be evaluated by a parallel prefix scan (Blelloch, 1990) in sequential rounds:
- the affine map ; composing two of them is again affine, which is all a scan needs
- associative but not commutative: order along the sequence matters
The real bottleneck is memory traffic. The expanded state has shape (B, L, D, N), times larger than the input. Mamba's kernel never writes it to GPU main memory (HBM): it loads , , , into on-chip SRAM, discretises and scans there (kernel fusion), and writes back only the output of shape (B, L, D). For the backward pass the states are recomputed rather than stored, so the layer's memory matches an optimised Transformer with FlashAttention. The paper reports the scan faster than FlashAttention-2 beyond sequence length 2K and 20–40× faster than a standard scan in PyTorch.
At a glance
- Training
- work, parallel scan over ; no term
- Generation
- per token with a constant-size state: no KV cache that grows with context
- Throughput
- 5× the generation throughput of a Transformer of similar size (paper, Section 4.5)
- Defaults
- state , expansion , conv width 4, rank , real diagonal
- Block params
- : in, out; two Mamba blocks ≈ one Transformer layer ()
- Lineage
- S4 → H3 → S6 (selective S4) → Mamba → Mamba-2 (SSD)
The Mamba block
Gu and Dao merge the H3 block (the basis of earlier SSM language models) and a gated MLP into one block. The input of width is projected to two branches of width . The main branch passes through a depthwise causal conv1d of width 4, a SiLU, and the selective SSM; the other branch is a SiLU gate. Their product is projected back to . Blocks are stacked with residual connections and normalisation (LayerNorm or RMSNorm), and nothing else: no attention and no separate MLP.
| part | shape | parameters |
|---|---|---|
| in_proj (x and z branches) | 768 → 2 × 1536 | 2,359,296 |
| conv1d (depthwise, width 4, bias) | 1536 × 4 + 1536 | 7,680 |
| x_proj (Δ rank 48, B, C) | 1536 → 48 + 16 + 16 | 122,880 |
| dt_proj (with bias) | 48 → 1536 | 75,264 |
| A_log, D | 1536 × 16, 1536 | 26,112 |
| out_proj | 1536 → 768 | 1,179,648 |
| total | 3,770,880 |
Worked example: five tokens through a selective gate
One channel, , , , values where tokens 2 and 5 are marked. The selective model uses on marked tokens and on filler; the LTI model uses for all, i.e. a constant gate .
Selective scan by hand (checked in Node with the figure's library)
- 1Steps: on marked and on filler tokens. Gates and .
- 2 (filler barely enters). (marked: almost a full overwrite).
- 3 and : the stored 1.0 survives two filler tokens. after the next marked token .
- 4LTI with constant gate 0.5: . After token 3 it has already lost the 1.0, because it cannot tell filler from content.
Results
Selective copying (Table 1): S4 with no gating reaches 18.3% accuracy; replacing S4 by the selective S6 lifts the same architecture to 97.0%; the full Mamba block with S6 reaches 99.8%, while the Mamba block with S4 inside gets 56.4%. Selection, not the gated architecture, does the work. Induction heads (Table 2): trained at length 256, Mamba solves the task perfectly and extrapolates to sequences of about one million tokens, 4000× longer, while no other method tested generalises beyond 2× its training length.
| model | Pile ppl ↓ | LAMBADA ppl ↓ | average acc ↑ |
|---|---|---|---|
| Pythia-2.8B | 6.73 | 5.04 | 59.1 |
| RWKV-3B | 7.00 | 5.24 | 59.6 |
| Mamba-2.8B | 6.22 | 4.23 | 63.3 |
| Pythia-6.9B | 6.51 | 4.45 | 61.7 |
The abstract's summary: "our Mamba-3B model outperforms Transformers of the same size and matches Transformers twice its size, both in pretraining and downstream evaluation". The paper also reports strong results on DNA and raw-audio modelling, where performance keeps improving with context up to million-length sequences.
Mamba-2 (Dao and Gu, "Transformers are SSMs", ICML 2024) restricts to a scalar times the identity per head and uses a larger head dimension. The resulting state space duality (SSD) shows that this SSM equals a form of masked linear attention, with a decay mask , which can be computed with matrix multiplications. Its core layer is 2–8× faster than Mamba's scan and allows recurrent states 8× the size of Mamba's or more with minimal slowdown.
1import math, torch, torch.nn as nn, torch.nn.functional as F23def selective_scan(x, delta, A, B, C, D):4 """Sequential reference. x, delta: (b, l, d); A: (d, n); B, C: (b, l, n); D: (d,)."""5 b, l, d = x.shape6 dA = torch.exp(delta.unsqueeze(-1) * A) # Abar_t = exp(delta_t A): (b, l, d, n)7 dBx = delta.unsqueeze(-1) * B.unsqueeze(2) * x.unsqueeze(-1) # Bbar_t x_t ~ delta_t B_t x_t (as in mamba_ssm's ref)8 h = x.new_zeros(b, d, A.shape[1])9 ys = []10 for t in range(l): # the fused kernel does this as a parallel scan in SRAM11 h = dA[:, t] * h + dBx[:, t]12 ys.append((h * C[:, t].unsqueeze(1)).sum(-1)) # y_t = C_t h_t13 return torch.stack(ys, dim=1) + x * D1415class MambaBlock(nn.Module):16 def __init__(self, d_model, d_state=16, d_conv=4, expand=2):17 super().__init__()18 di, self.n = expand * d_model, d_state19 self.r = math.ceil(d_model / 16) # rank of the delta projection20 self.in_proj = nn.Linear(d_model, 2 * di, bias=False)21 self.conv1d = nn.Conv1d(di, di, d_conv, groups=di, padding=d_conv - 1)22 self.x_proj = nn.Linear(di, self.r + 2 * d_state, bias=False)23 self.dt_proj = nn.Linear(self.r, di)24 dt = torch.exp(torch.rand(di) * (math.log(0.1) - math.log(1e-3)) + math.log(1e-3))25 with torch.no_grad():26 self.dt_proj.bias.copy_(dt + torch.log(-torch.expm1(-dt))) # inverse softplus: delta starts in [1e-3, 0.1]27 self.A_log = nn.Parameter(torch.log(torch.arange(1, d_state + 1, dtype=torch.float32).repeat(di, 1)))28 self.D = nn.Parameter(torch.ones(di))29 self.out_proj = nn.Linear(di, d_model, bias=False)3031 def forward(self, u): # u: (b, l, d_model)32 l = u.shape[1]33 x, z = self.in_proj(u).chunk(2, dim=-1)34 x = F.silu(self.conv1d(x.transpose(1, 2))[..., :l].transpose(1, 2)) # causal depthwise conv35 dt, B, C = self.x_proj(x).split([self.r, self.n, self.n], dim=-1) # input-dependent delta, B, C36 delta = F.softplus(self.dt_proj(dt))37 A = -torch.exp(self.A_log) # S4D-Real init: A_n = -(n + 1)38 y = selective_scan(x, delta, A, B, C, self.D)39 return self.out_proj(y * F.silu(z)) # SiLU gate branch4041blk = MambaBlock(768)42print(sum(p.numel() for p in blk.parameters())) # 377088043print(blk(torch.randn(2, 64, 768)).shape) # torch.Size([2, 64, 768])4445# Fused, hardware-aware version (pip install mamba-ssm; needs a CUDA GPU):46# from mamba_ssm import Mamba47# m = Mamba(d_model=768, d_state=16, d_conv=4, expand=2).to("cuda")48# y = m(torch.randn(2, 64, 768, device="cuda"))Use when
- +Linear-time training and constant-memory O(1)-per-token generation: no growing KV cache
- +Content-aware: Δ, B, C let the model skip filler, store key tokens and reset at boundaries
- +Matches Transformers about twice its size at 3B scale on the paper's zero-shot suite; 5× generation throughput
- +Simple homogeneous block (no attention, no separate MLP)
Avoid when
- −No convolution form: needs a custom fused scan kernel to be fast (CUDA-specific implementations)
- −Fixed-size state limits exact retrieval and copying of long spans compared with attention
- −Sequential dependence remains at inference: tokens of one sequence are still processed one at a time
- −Younger ecosystem; best results often come from hybrids with attention layers
Check yourself
In Mamba's Theorem 1 setting (N = 1, A = −1, B = 1, Δ = softplus(z)), what happens when a token produces a very large Δ?
Check yourself
Why does Mamba need a parallel scan instead of the FFT convolution used to train S4?
Remember
- ✓Selective SSM: , , , .
- ✓With one state it is a gated RNN, , : large stores, small ignores.
- ✓Input dependence removes the convolution view; a fused, SRAM-resident parallel scan with recomputation keeps training linear-time and memory-efficient.
- ✓Mamba block: in_proj to , causal conv1d + SiLU + S6 on one branch, SiLU gate on the other, out_proj; params, , .
- ✓Mamba-2.8B averages 63.3 zero-shot vs 59.1 for Pythia-2.8B and 61.7 for Pythia-6.9B (Table 3); Mamba-2 (SSD) recasts the layer as masked linear attention and is 2–8× faster.