Definition
DropConnect is a regularizer that, during training, multiplies a layer's weight matrix by a random binary mask: with each independently. Dropout removes a unit together with all of its connections; DropConnect removes single connections.
Wan, Zeiler, Zhang, LeCun and Fergus introduced it (2013, "Regularization of neural networks using DropConnect", ICML) as a generalization of Dropout. Dropping input unit is the same as zeroing column of ; DropConnect lets every entry of be dropped on its own. A layer with inputs and outputs then has possible masks instead of , so the implicit ensemble is far larger. They applied it to the fully connected layer on top of convolutional feature extractors, proved a Rademacher-complexity generalization bound, and reported state-of-the-art results on several image benchmarks at the time by averaging a few DropConnect-trained models.
The idea lived on most prominently in language modelling. Merity, Keskar and Socher's AWD-LSTM (2017, "Regularizing and optimizing LSTM language models") applies DropConnect to the LSTM's recurrent weight matrices, which ordinary dropout cannot touch without destroying the hidden state's memory.
Formula
- the layer's input (activations of the previous layer)
- binary mask with the same shape as W, redrawn for every example in Wan et al.
- keep probability (); Wan et al. call it and use 0.5
- pre-activation of output unit i; the layer outputs r = a(u)
The gradient also flows through the mask: , so a dropped weight gets no update from that example, while the rest of the row and column still learn. Wan et al. did not rescale during training; frameworks that implement it with F.dropout on the weight tensor rescale the kept weights by instead, which moves the factor from test time to training time exactly as inverted dropout does.
At a glance
- Masks
- weights , not units; masks per layer
- Single unit
- for one output unit, masking weight or input gives the same distribution of ; the difference is that DropConnect's masks are independent across output units
- Typical q
- 0.5 in Wan et al.; AWD-LSTM drops 0.5 of the hidden-to-hidden weights on Penn Treebank
- Inference
- mean inference (scale weights by q) or Gaussian moment matching with sampling
- Cost
- a mask per example: much more memory and randomness than dropout's
- PyTorch
- no built-in layer; use
F.dropouton the weight, ortorch.nn.utils.parametrize
Inference: why the mean is not enough
At test time we want the average over masks, . The cheap option, mean inference, plugs in the mean pre-activation: . This is dropout's weight-scaling rule. It is exact for a linear layer but biased after a nonlinearity, and the bias is largest when the unit sits near its kink. Wan et al. instead use moment matching: is a sum of many independent terms, so by the central limit theorem it is approximately Gaussian with the mean and variance above. They draw samples from that Gaussian per unit, average , and feed the result to the next layer. For ReLU the Gaussian average has a closed form:
- standard normal CDF and density
- the extra output from masks that push above 0; it is largest at (), exactly where mean inference is furthest off
Worked example: three weights, all eight masks
Take , and . The products are , so is a sum of a random subset of .
| quantity | value | how |
|---|---|---|
| values of | one per mask | |
| exact | ||
| moment matching | ||
| mean inference | ignores the spread of u |
With only three terms the Gaussian is crude and slightly overshoots (1.303 vs 1.25); with 64 inputs in the figure the gap is 0.006. Mean inference is off by 0.25 here, and it is the mean, not the Gaussian, that gets worse as the unit moves toward its kink.
Weight-dropped LSTM (AWD-LSTM)
An LSTM computes its gates from . Dropout with a fresh mask on at every step corrupts the memory the network is supposed to carry. AWD-LSTM applies DropConnect to the hidden-to-hidden matrices instead: one mask is drawn per forward pass, shared across all time steps and the whole batch, and the masked is handed to an unmodified LSTM. Because the recurrence itself is untouched, the fast fused cuDNN kernel still works. Combined with variational dropout on inputs and outputs, embedding dropout, weight tying and NT-ASGD, it set state-of-the-art word-level perplexities on Penn Treebank and WikiText-2 in 2017.
One AWD-LSTM forward pass
- 1Sample with the shape of (e.g. ).
- 2Form once (inverted scaling, so evaluation uses unchanged).
- 3Run the whole sequence with : every time step and every sequence in the batch uses the same dropped connections.
- 4Backward: gradients reach only the kept entries of ; next batch, a new mask.
1import math, torch, torch.nn as nn, torch.nn.functional as F23class DropConnectLinear(nn.Linear):4 """One weight mask per forward pass, shared by the batch (as in AWD-LSTM)."""5 def __init__(self, n_in, n_out, p=0.5):6 super().__init__(n_in, n_out)7 self.p = p8 def forward(self, x):9 w = F.dropout(self.weight, self.p, self.training) # inverted: kept weights / (1-p)10 return F.linear(x, w, self.bias)1112def dropconnect_per_example(x, weight, p):13 """Wan et al.'s version: an independent (out, in) mask for every example."""14 mask = (torch.rand(x.shape[0], *weight.shape, device=x.device) > p).to(x.dtype)15 return torch.einsum("noi,ni->no", mask * weight / (1 - p), x) # memory: N*out*in1617@torch.no_grad()18def moment_matching_relu(x, weight, bias, p):19 """E[ReLU(u)] under a Gaussian fit to u (inverted scaling, so the mean is W x)."""20 q = 1 - p21 mu = F.linear(x, weight, bias)22 var = (p / q) * F.linear(x * x, weight * weight) # q(1-q) (W∘W)(x∘x) / q^223 s = var.clamp_min(1e-12).sqrt()24 z = mu / s25 Phi = 0.5 * (1 + torch.erf(z / math.sqrt(2)))26 phi = torch.exp(-0.5 * z * z) / math.sqrt(2 * math.pi)27 return mu * Phi + s * phi2829# Weight-dropped LSTM: mask the recurrent weights once per forward pass.30# Simplest version: drop the hidden-to-hidden weights of an LSTMCell and unroll it.31cell = nn.LSTMCell(400, 1150)32def wd_lstm(xs, h, c, p=0.5, training=True):33 w_hh = F.dropout(cell.weight_hh, p, training) # same mask for all time steps34 outs = []35 for x in xs: # xs: (T, N, 400)36 gates = F.linear(x, cell.weight_ih, cell.bias_ih) + F.linear(h, w_hh, cell.bias_hh)37 i, f, g, o = gates.chunk(4, dim=1)38 c = torch.sigmoid(f) * c + torch.sigmoid(i) * torch.tanh(g)39 h = torch.sigmoid(o) * torch.tanh(c)40 outs.append(h)41 return torch.stack(outs), (h, c)Use when
- +Much richer noise than dropout: masks, independent per output unit
- +Regularizes recurrent weights without corrupting the hidden state (AWD-LSTM)
- +Batch-shared version is a one-line change and keeps fused kernels (cuDNN LSTM) usable
- +Moment-matching inference gives a better test-time approximation than the mean
Avoid when
- −Per-example masks cost memory and break the single matrix multiply
- −Better inference needs sampling or closed forms per unit, not just a weight scale
- −No built-in PyTorch layer; easy to implement subtly wrong
- −Rarely beats dropout plus weight decay in CNNs and Transformers
Check yourself
A ReLU unit under DropConnect has mean pre-activation μ = −0.4 and standard deviation σ = 0.8. What does mean inference output, and is it too high or too low?
Check yourself
Why does AWD-LSTM apply DropConnect to the hidden-to-hidden weights rather than dropout to the hidden state?
Remember
- ✓DropConnect masks weights: , ; dropout masks units (whole columns of ).
- ✓ and ; for one unit the distribution is the same as dropout on its inputs.
- ✓Mean inference is biased after a nonlinearity; Wan et al. fit a Gaussian to and average over samples.
- ✓Per-example masks are costly; practical versions share one mask per batch.
- ✓AWD-LSTM's weight-dropped LSTM masks the recurrent matrices once per forward pass.