DropConnect

Randomly zero individual weights instead of whole units. Each output sees its own mask, and test time needs more than the mean: Wan et al. match a Gaussian to each pre-activation. AWD-LSTM applies it to recurrent weights.

11 min read885 wordsadvanced0 viewsupdated 2026-10-11

Definition

DropConnect is a regularizer that, during training, multiplies a layer's weight matrix by a random binary mask: r=a((M⊙W) v)r=a\big((M\odot W)\,v\big) with each Mij∼Bernoulli(q)M_{ij}\sim\mathrm{Bernoulli}(q) independently. Dropout removes a unit together with all of its connections; DropConnect removes single connections.

Wan, Zeiler, Zhang, LeCun and Fergus introduced it (2013, "Regularization of neural networks using DropConnect", ICML) as a generalization of Dropout. Dropping input unit vjv_j is the same as zeroing column jj of WW; DropConnect lets every entry of WW be dropped on its own. A layer with nn inputs and dd outputs then has 2nd2^{nd} possible masks instead of 2n2^n, so the implicit ensemble is far larger. They applied it to the fully connected layer on top of convolutional feature extractors, proved a Rademacher-complexity generalization bound, and reported state-of-the-art results on several image benchmarks at the time by averaging a few DropConnect-trained models.

The idea lived on most prominently in language modelling. Merity, Keskar and Socher's AWD-LSTM (2017, "Regularizing and optimizing LSTM language models") applies DropConnect to the LSTM's recurrent weight matrices, which ordinary dropout cannot touch without destroying the hidden state's memory.

Formula

u=(M⊙W) v,Mij∼Bernoulli(q),E[ui]=q∑jWijvj,Var⁡[ui]=q(1−q)∑jWij2vj2u=(M\odot W)\,v,\quad M_{ij}\sim\mathrm{Bernoulli}(q),\qquad \mathbb E[u_i]=q\sum_j W_{ij}v_j,\qquad \operatorname{Var}[u_i]=q(1-q)\sum_j W_{ij}^2v_j^2
vv
the layer's input (activations of the previous layer)
MM
binary mask with the same shape as W, redrawn for every example in Wan et al.
qq
keep probability (q=1−pq=1-p); Wan et al. call it pp and use 0.5
uiu_i
pre-activation of output unit i; the layer outputs r = a(u)

The gradient also flows through the mask: ∂L/∂Wij=Mij δi vj\partial L/\partial W_{ij}=M_{ij}\,\delta_i\,v_j, so a dropped weight gets no update from that example, while the rest of the row and column still learn. Wan et al. did not rescale during training; frameworks that implement it with F.dropout on the weight tensor rescale the kept weights by 1/q1/q instead, which moves the factor qq from test time to training time exactly as inverted dropout does.

At a glance

Masks
weights WijW_{ij}, not units; 2nd2^{nd} masks per d×nd\times n layer
Single unit
for one output unit, masking weight WijW_{ij} or input vjv_j gives the same distribution of uiu_i; the difference is that DropConnect's masks are independent across output units
Typical q
0.5 in Wan et al.; AWD-LSTM drops 0.5 of the hidden-to-hidden weights on Penn Treebank
Inference
mean inference (scale weights by q) or Gaussian moment matching with sampling
Cost
a d×nd\times n mask per example: much more memory and randomness than dropout's nn
PyTorch
no built-in layer; use F.dropout on the weight, or torch.nn.utils.parametrize

Inference: why the mean is not enough

At test time we want the average over masks, EM[a((M⊙W)v)]\mathbb E_M\big[a\big((M\odot W)v\big)\big]. The cheap option, mean inference, plugs in the mean pre-activation: a(E[u])=a(qWv)a\big(\mathbb E[u]\big)=a(qWv). This is dropout's weight-scaling rule. It is exact for a linear layer but biased after a nonlinearity, and the bias is largest when the unit sits near its kink. Wan et al. instead use moment matching: uiu_i is a sum of many independent terms, so by the central limit theorem it is approximately Gaussian with the mean and variance above. They draw ZZ samples from that Gaussian per unit, average a(u)a(u), and feed the result to the next layer. For ReLU the Gaussian average has a closed form:

E[ReLU(u)]≈μ Φ ⁣(μσ)+σ φ ⁣(μσ),μ=q Wv,  σ2=q(1−q) (W⊙W)(v⊙v)\mathbb E\big[\mathrm{ReLU}(u)\big]\approx\mu\,\Phi\!\Big(\frac{\mu}{\sigma}\Big)+\sigma\,\varphi\!\Big(\frac{\mu}{\sigma}\Big),\qquad \mu=q\,Wv,\ \ \sigma^2=q(1-q)\,(W\odot W)(v\odot v)
Φ, φ\Phi,\ \varphi
standard normal CDF and density
σ φ(μ/σ)\sigma\,\varphi(\mu/\sigma)
the extra output from masks that push uu above 0; it is largest at μ=0\mu=0 (0.399 σ0.399\,\sigma), exactly where mean inference ReLU(μ)\mathrm{ReLU}(\mu) is furthest off
W: 5 outputs × 8 inputsdropout: drop inputs vᵢa whole column goes · 2⁸ = 256 masksDropConnect: drop weights Wᵢⱼany entry goes · 2⁴⁰ ≈ 1.1·10¹² masksu = Σ Mⱼ Wⱼ vⱼ + b, 64 inputs, 4000 masksReLU = 0 hereμ-3.190.113.41μ = 0.108 · σ = 0.824 · on in 55% of maskstest-time output E[ReLU(u + b)]:exact: mean over masks0.380moment matching N(μ, σ²)0.386mean inference ReLU(μ)0.108■ kept weight ⬚ dropped ━ Gaussian with the same mean and variance
q = 1 − p kept; no rescaling (Wan et al.)
fig.Left: one random mask on a 5×8 weight matrix. Dropout removes whole input columns (2⁸ = 256 possible masks); DropConnect removes single entries (2⁴⁰ ≈ 1.1·10¹² masks). Right: one ReLU unit with 64 inputs, its pre-activation over 4000 random weight masks (bars) and the Gaussian with the same mean and variance (violet). At the default p = 0.5, b = 0: μ = 0.108 and σ = 0.824, and the unit is on in 55% of masks. The exact test-time output, the average ReLU over the masks, is 0.380; Gaussian moment matching gives 0.386; mean inference, ReLU(μ), gives 0.108, less than a third. Move b to +1 and the unit is on in 91% of masks: all three agree within 0.03 (1.138, 1.143, 1.108). Move b to −0.5: mean inference outputs 0 while the true average is 0.166.

Worked example: three weights, all eight masks

Take W=(1, −2, 0.5)W=(1,\,-2,\,0.5), v=(2, 1, 4)v=(2,\,1,\,4) and q=0.5q=0.5. The products are Wjvj=(2, −2, 2)W_jv_j=(2,\,-2,\,2), so uu is a sum of a random subset of {2,−2,2}\{2,-2,2\}.

quantityvaluehow
values of uu0, 2, −2, 0, 2, 4, 0, 20,\,2,\,-2,\,0,\,2,\,4,\,0,\,2one per mask (M1,M2,M3)∈{0,1}3(M_1,M_2,M_3)\in\{0,1\}^3
μ=E[u]\mu=\mathbb E[u]1.01.0q∑jWjvj=0.5⋅2q\sum_j W_jv_j=0.5\cdot2
σ2=Var⁡[u]\sigma^2=\operatorname{Var}[u]3.03.0q(1−q)∑j(Wjvj)2=0.25⋅12q(1-q)\sum_j (W_jv_j)^2=0.25\cdot12
exact E[ReLU(u)]\mathbb E[\mathrm{ReLU}(u)]1.251.25(0+2+0+0+2+4+0+2)/8(0+2+0+0+2+4+0+2)/8
moment matching1.3031.3031⋅Φ(0.577)+1.732⋅φ(0.577)=0.718+0.5851\cdot\Phi(0.577)+1.732\cdot\varphi(0.577)=0.718+0.585
mean inference ReLU(μ)\mathrm{ReLU}(\mu)1.01.0ignores the spread of u
All 2³ masks are equally likely (checked in Python)

With only three terms the Gaussian is crude and slightly overshoots (1.303 vs 1.25); with 64 inputs in the figure the gap is 0.006. Mean inference is off by 0.25 here, and it is the mean, not the Gaussian, that gets worse as the unit moves toward its kink.

Weight-dropped LSTM (AWD-LSTM)

An LSTM computes its gates from Wxt+Uht−1W x_t + U h_{t-1}. Dropout with a fresh mask on ht−1h_{t-1} at every step corrupts the memory the network is supposed to carry. AWD-LSTM applies DropConnect to the hidden-to-hidden matrices U=[Ui,Uf,Uo,Uc]U=[U^i,U^f,U^o,U^c] instead: one mask is drawn per forward pass, shared across all time steps and the whole batch, and the masked UU is handed to an unmodified LSTM. Because the recurrence itself is untouched, the fast fused cuDNN kernel still works. Combined with variational dropout on inputs and outputs, embedding dropout, weight tying and NT-ASGD, it set state-of-the-art word-level perplexities on Penn Treebank and WikiText-2 in 2017.

One AWD-LSTM forward pass

  1. 1Sample M∼Bernoulli(1−p)M\sim\mathrm{Bernoulli}(1-p) with the shape of UU (e.g. p=0.5p=0.5).
  2. 2Form U~=M⊙U/(1−p)\tilde U=M\odot U/(1-p) once (inverted scaling, so evaluation uses UU unchanged).
  3. 3Run the whole sequence with U~\tilde U: every time step and every sequence in the batch uses the same dropped connections.
  4. 4Backward: gradients reach only the kept entries of UU; next batch, a new mask.
python
1import math, torch, torch.nn as nn, torch.nn.functional as F
2
3class DropConnectLinear(nn.Linear):
4 """One weight mask per forward pass, shared by the batch (as in AWD-LSTM)."""
5 def __init__(self, n_in, n_out, p=0.5):
6 super().__init__(n_in, n_out)
7 self.p = p
8 def forward(self, x):
9 w = F.dropout(self.weight, self.p, self.training) # inverted: kept weights / (1-p)
10 return F.linear(x, w, self.bias)
11
12def dropconnect_per_example(x, weight, p):
13 """Wan et al.'s version: an independent (out, in) mask for every example."""
14 mask = (torch.rand(x.shape[0], *weight.shape, device=x.device) > p).to(x.dtype)
15 return torch.einsum("noi,ni->no", mask * weight / (1 - p), x) # memory: N*out*in
16
17@torch.no_grad()
18def moment_matching_relu(x, weight, bias, p):
19 """E[ReLU(u)] under a Gaussian fit to u (inverted scaling, so the mean is W x)."""
20 q = 1 - p
21 mu = F.linear(x, weight, bias)
22 var = (p / q) * F.linear(x * x, weight * weight) # q(1-q) (W∘W)(x∘x) / q^2
23 s = var.clamp_min(1e-12).sqrt()
24 z = mu / s
25 Phi = 0.5 * (1 + torch.erf(z / math.sqrt(2)))
26 phi = torch.exp(-0.5 * z * z) / math.sqrt(2 * math.pi)
27 return mu * Phi + s * phi
28
29# Weight-dropped LSTM: mask the recurrent weights once per forward pass.
30# Simplest version: drop the hidden-to-hidden weights of an LSTMCell and unroll it.
31cell = nn.LSTMCell(400, 1150)
32def wd_lstm(xs, h, c, p=0.5, training=True):
33 w_hh = F.dropout(cell.weight_hh, p, training) # same mask for all time steps
34 outs = []
35 for x in xs: # xs: (T, N, 400)
36 gates = F.linear(x, cell.weight_ih, cell.bias_ih) + F.linear(h, w_hh, cell.bias_hh)
37 i, f, g, o = gates.chunk(4, dim=1)
38 c = torch.sigmoid(f) * c + torch.sigmoid(i) * torch.tanh(g)
39 h = torch.sigmoid(o) * torch.tanh(c)
40 outs.append(h)
41 return torch.stack(outs), (h, c)
DropConnect layers in PyTorch: batch-shared mask, per-example masks, and moment-matching inference

Use when

  • +Much richer noise than dropout: 2nd2^{nd} masks, independent per output unit
  • +Regularizes recurrent weights without corrupting the hidden state (AWD-LSTM)
  • +Batch-shared version is a one-line change and keeps fused kernels (cuDNN LSTM) usable
  • +Moment-matching inference gives a better test-time approximation than the mean

Avoid when

  • −Per-example masks cost memory and break the single matrix multiply
  • −Better inference needs sampling or closed forms per unit, not just a weight scale
  • −No built-in PyTorch layer; easy to implement subtly wrong
  • −Rarely beats dropout plus weight decay in CNNs and Transformers

Check yourself

A ReLU unit under DropConnect has mean pre-activation μ = −0.4 and standard deviation σ = 0.8. What does mean inference output, and is it too high or too low?

Check yourself

Why does AWD-LSTM apply DropConnect to the hidden-to-hidden weights rather than dropout to the hidden state?

Remember

  • ✓DropConnect masks weights: u=(M⊙W)vu=(M\odot W)v, Mij∼Bernoulli(q)M_{ij}\sim\mathrm{Bernoulli}(q); dropout masks units (whole columns of WW).
  • ✓E[u]=qWv\mathbb E[u]=qWv and Var⁡[u]=q(1−q)(W⊙W)(v⊙v)\operatorname{Var}[u]=q(1-q)(W\odot W)(v\odot v); for one unit the distribution is the same as dropout on its inputs.
  • ✓Mean inference a(qWv)a(qWv) is biased after a nonlinearity; Wan et al. fit a Gaussian to uu and average a(u)a(u) over samples.
  • ✓Per-example masks are costly; practical versions share one mask per batch.
  • ✓AWD-LSTM's weight-dropped LSTM masks the recurrent matrices once per forward pass.
← back to Deep Learning roadmap