# DropConnect

> Randomly zero individual weights instead of whole units. Each output sees its own mask, and test time needs more than the mean: Wan et al. match a Gaussian to each pre-activation. AWD-LSTM applies it to recurrent weights.

Source: https://www.mlfoundry.in/learn/dropconnect  
Section: Deep Learning › Regularization & Normalization  
Level: advanced · 11 min read · updated 2026-10-11

**DropConnect:** is a regularizer that, during training, multiplies a layer's weight matrix by a random binary mask: $r=a\big((M\odot W)\,v\big)$ with each $M_{ij}\sim\mathrm{Bernoulli}(q)$ independently. Dropout removes a unit together with all of its connections; DropConnect removes single connections.

Wan, Zeiler, Zhang, LeCun and Fergus introduced it (2013, "Regularization of neural networks using DropConnect", ICML) as a generalization of **Dropout**. Dropping input unit $v_j$ is the same as zeroing column $j$ of $W$; DropConnect lets every entry of $W$ be dropped on its own. A layer with $n$ inputs and $d$ outputs then has $2^{nd}$ possible masks instead of $2^n$, so the implicit ensemble is far larger. They applied it to the fully connected layer on top of convolutional feature extractors, proved a Rademacher-complexity generalization bound, and reported state-of-the-art results on several image benchmarks at the time by averaging a few DropConnect-trained models.

The idea lived on most prominently in language modelling. Merity, Keskar and Socher's AWD-LSTM (2017, "Regularizing and optimizing LSTM language models") applies DropConnect to the LSTM's **recurrent** weight matrices, which ordinary dropout cannot touch without destroying the hidden state's memory.

## Formula

$$
u=(M\odot W)\,v,\quad M_{ij}\sim\mathrm{Bernoulli}(q),\qquad \mathbb E[u_i]=q\sum_j W_{ij}v_j,\qquad \operatorname{Var}[u_i]=q(1-q)\sum_j W_{ij}^2v_j^2
$$

- $v$: the layer's input (activations of the previous layer)
- $M$: binary mask with the same shape as W, redrawn for every example in Wan et al.
- $q$: keep probability ($q=1-p$); Wan et al. call it $p$ and use 0.5
- $u_i$: pre-activation of output unit i; the layer outputs r = a(u)

The gradient also flows through the mask: $\partial L/\partial W_{ij}=M_{ij}\,\delta_i\,v_j$, so a dropped weight gets no update from that example, while the rest of the row and column still learn. Wan et al. did not rescale during training; frameworks that implement it with `F.dropout` on the weight tensor rescale the kept weights by $1/q$ instead, which moves the factor $q$ from test time to training time exactly as inverted dropout does.

**At a glance**

- **Masks:** weights $W_{ij}$, not units; $2^{nd}$ masks per $d\times n$ layer
- **Single unit:** for one output unit, masking weight $W_{ij}$ or input $v_j$ gives the same distribution of $u_i$; the difference is that DropConnect's masks are independent across output units
- **Typical q:** 0.5 in Wan et al.; AWD-LSTM drops 0.5 of the hidden-to-hidden weights on Penn Treebank
- **Inference:** mean inference (scale weights by q) or Gaussian moment matching with sampling
- **Cost:** a $d\times n$ mask per example: much more memory and randomness than dropout's $n$
- **PyTorch:** no built-in layer; use `F.dropout` on the weight, or `torch.nn.utils.parametrize`

## Inference: why the mean is not enough

At test time we want the average over masks, $\mathbb E_M\big[a\big((M\odot W)v\big)\big]$. The cheap option, **mean inference**, plugs in the mean pre-activation: $a\big(\mathbb E[u]\big)=a(qWv)$. This is dropout's weight-scaling rule. It is exact for a linear layer but biased after a nonlinearity, and the bias is largest when the unit sits near its kink. Wan et al. instead use **moment matching**: $u_i$ is a sum of many independent terms, so by the central limit theorem it is approximately Gaussian with the mean and variance above. They draw $Z$ samples from that Gaussian per unit, average $a(u)$, and feed the result to the next layer. For ReLU the Gaussian average has a closed form:

$$
\mathbb E\big[\mathrm{ReLU}(u)\big]\approx\mu\,\Phi\!\Big(\frac{\mu}{\sigma}\Big)+\sigma\,\varphi\!\Big(\frac{\mu}{\sigma}\Big),\qquad \mu=q\,Wv,\ \ \sigma^2=q(1-q)\,(W\odot W)(v\odot v)
$$

- $\Phi,\ \varphi$: standard normal CDF and density
- $\sigma\,\varphi(\mu/\sigma)$: the extra output from masks that push $u$ above 0; it is largest at $\mu=0$ ($0.399\,\sigma$), exactly where mean inference $\mathrm{ReLU}(\mu)$ is furthest off

> Figure: Left: one random mask on a 5×8 weight matrix. Dropout removes whole input columns (2⁸ = 256 possible masks); DropConnect removes single entries (2⁴⁰ ≈ 1.1·10¹² masks). Right: one ReLU unit with 64 inputs, its pre-activation over 4000 random weight masks (bars) and the Gaussian with the same mean and variance (violet). At the default p = 0.5, b = 0: μ = 0.108 and σ = 0.824, and the unit is on in 55% of masks. The exact test-time output, the average ReLU over the masks, is 0.380; Gaussian moment matching gives 0.386; mean inference, ReLU(μ), gives 0.108, less than a third. Move b to +1 and the unit is on in 91% of masks: all three agree within 0.03 (1.138, 1.143, 1.108). Move b to −0.5: mean inference outputs 0 while the true average is 0.166.

## Worked example: three weights, all eight masks

Take $W=(1,\,-2,\,0.5)$, $v=(2,\,1,\,4)$ and $q=0.5$. The products are $W_jv_j=(2,\,-2,\,2)$, so $u$ is a sum of a random subset of $\{2,-2,2\}$.

All 2³ masks are equally likely (checked in Python)

| quantity | value | how |
| --- | --- | --- |
| values of $u$ | $0,\,2,\,-2,\,0,\,2,\,4,\,0,\,2$ | one per mask $(M_1,M_2,M_3)\in\{0,1\}^3$ |
| $\mu=\mathbb E[u]$ | $1.0$ | $q\sum_j W_jv_j=0.5\cdot2$ |
| $\sigma^2=\operatorname{Var}[u]$ | $3.0$ | $q(1-q)\sum_j (W_jv_j)^2=0.25\cdot12$ |
| exact $\mathbb E[\mathrm{ReLU}(u)]$ | $1.25$ | $(0+2+0+0+2+4+0+2)/8$ |
| moment matching | $1.303$ | $1\cdot\Phi(0.577)+1.732\cdot\varphi(0.577)=0.718+0.585$ |
| mean inference $\mathrm{ReLU}(\mu)$ | $1.0$ | ignores the spread of u |

With only three terms the Gaussian is crude and slightly overshoots (1.303 vs 1.25); with 64 inputs in the figure the gap is 0.006. Mean inference is off by 0.25 here, and it is the mean, not the Gaussian, that gets worse as the unit moves toward its kink.

## Weight-dropped LSTM (AWD-LSTM)

An LSTM computes its gates from $W x_t + U h_{t-1}$. Dropout with a fresh mask on $h_{t-1}$ at every step corrupts the memory the network is supposed to carry. AWD-LSTM applies DropConnect to the hidden-to-hidden matrices $U=[U^i,U^f,U^o,U^c]$ instead: one mask is drawn **per forward pass**, shared across all time steps and the whole batch, and the masked $U$ is handed to an unmodified LSTM. Because the recurrence itself is untouched, the fast fused cuDNN kernel still works. Combined with variational dropout on inputs and outputs, embedding dropout, weight tying and NT-ASGD, it set state-of-the-art word-level perplexities on Penn Treebank and WikiText-2 in 2017.

**One AWD-LSTM forward pass**

1. Sample $M\sim\mathrm{Bernoulli}(1-p)$ with the shape of $U$ (e.g. $p=0.5$).
2. Form $\tilde U=M\odot U/(1-p)$ once (inverted scaling, so evaluation uses $U$ unchanged).
3. Run the whole sequence with $\tilde U$: every time step and every sequence in the batch uses the same dropped connections.
4. Backward: gradients reach only the kept entries of $U$; next batch, a new mask.

```python
import math, torch, torch.nn as nn, torch.nn.functional as F

class DropConnectLinear(nn.Linear):
    """One weight mask per forward pass, shared by the batch (as in AWD-LSTM)."""
    def __init__(self, n_in, n_out, p=0.5):
        super().__init__(n_in, n_out)
        self.p = p
    def forward(self, x):
        w = F.dropout(self.weight, self.p, self.training)   # inverted: kept weights / (1-p)
        return F.linear(x, w, self.bias)

def dropconnect_per_example(x, weight, p):
    """Wan et al.'s version: an independent (out, in) mask for every example."""
    mask = (torch.rand(x.shape[0], *weight.shape, device=x.device) > p).to(x.dtype)
    return torch.einsum("noi,ni->no", mask * weight / (1 - p), x)   # memory: N*out*in

@torch.no_grad()
def moment_matching_relu(x, weight, bias, p):
    """E[ReLU(u)] under a Gaussian fit to u (inverted scaling, so the mean is W x)."""
    q = 1 - p
    mu = F.linear(x, weight, bias)
    var = (p / q) * F.linear(x * x, weight * weight)             # q(1-q) (W∘W)(x∘x) / q^2
    s = var.clamp_min(1e-12).sqrt()
    z = mu / s
    Phi = 0.5 * (1 + torch.erf(z / math.sqrt(2)))
    phi = torch.exp(-0.5 * z * z) / math.sqrt(2 * math.pi)
    return mu * Phi + s * phi

# Weight-dropped LSTM: mask the recurrent weights once per forward pass.
# Simplest version: drop the hidden-to-hidden weights of an LSTMCell and unroll it.
cell = nn.LSTMCell(400, 1150)
def wd_lstm(xs, h, c, p=0.5, training=True):
    w_hh = F.dropout(cell.weight_hh, p, training)           # same mask for all time steps
    outs = []
    for x in xs:                                            # xs: (T, N, 400)
        gates = F.linear(x, cell.weight_ih, cell.bias_ih) + F.linear(h, w_hh, cell.bias_hh)
        i, f, g, o = gates.chunk(4, dim=1)
        c = torch.sigmoid(f) * c + torch.sigmoid(i) * torch.tanh(g)
        h = torch.sigmoid(o) * torch.tanh(c)
        outs.append(h)
    return torch.stack(outs), (h, c)
```

DropConnect layers in PyTorch: batch-shared mask, per-example masks, and moment-matching inference

> **Per-example masks are expensive:** Wan et al.'s formulation draws a $d\times n$ mask for every example. For a $4096\times4096$ layer and a batch of 128 that is about 2.1 billion mask entries per step, and the matrix product can no longer be one GEMM because every example has its own weights. Their implementation packed masks into bits and used custom GPU kernels. In practice almost everyone shares one mask per mini-batch, as AWD-LSTM does; this is cheap but adds correlated noise across the batch, so the regularizing effect per step is smaller.

> **Dropout and DropConnect look the same to one unit:** For a single output unit $i$, "drop input $j$" and "drop weight $W_{ij}$" are the same event, so the histogram in the figure would be identical under dropout on the inputs. The difference is across units: dropout makes every output lose input $j$ together, while DropConnect lets each output lose a different subset. The mean-inference bias in the figure is therefore also why dropout's weight-scaling rule is only approximate; Wang and Manning's "fast dropout" (2013) used the same Gaussian approximation for dropout itself.

> **Where it is used today:** DropConnect is rarely used in convolutional or Transformer layers. Its main practical home is recurrent weight matrices (AWD-LSTM and its descendants), and the same name is used for dropping whole residual branches in some codebases (EfficientNet's `drop_connect` is actually **Stochastic depth**, not Wan et al.'s weight masking).

**Strengths**

- Much richer noise than dropout: $2^{nd}$ masks, independent per output unit
- Regularizes recurrent weights without corrupting the hidden state (AWD-LSTM)
- Batch-shared version is a one-line change and keeps fused kernels (cuDNN LSTM) usable
- Moment-matching inference gives a better test-time approximation than the mean

**Limitations**

- Per-example masks cost memory and break the single matrix multiply
- Better inference needs sampling or closed forms per unit, not just a weight scale
- No built-in PyTorch layer; easy to implement subtly wrong
- Rarely beats dropout plus weight decay in CNNs and Transformers

**Question:** A ReLU unit under DropConnect has mean pre-activation μ = −0.4 and standard deviation σ = 0.8. What does mean inference output, and is it too high or too low?

- A. 0, too low: the masks that push u above 0 give a positive average
- B. −0.4, exact
- C. 0, exact, because the mean is negative
- D. 0.4, too high

**Answer:** A. 0, too low: the masks that push u above 0 give a positive average — Mean inference outputs $\mathrm{ReLU}(-0.4)=0$. The true average counts every mask with $u>0$; the Gaussian estimate is $-0.4\,\Phi(-0.5)+0.8\,\varphi(-0.5)=-0.123+0.282=0.158$. The figure shows the same thing at $b=-0.5$: mean inference 0, exact 0.166.

**Question:** Why does AWD-LSTM apply DropConnect to the hidden-to-hidden weights rather than dropout to the hidden state?

- A. Dropping hidden units with a new mask each step corrupts the memory; masking U once per pass leaves the recurrence intact and keeps cuDNN usable
- B. DropConnect is cheaper than dropout per element
- C. Dropout cannot be applied to LSTMs in PyTorch
- D. It removes the need for any other dropout

**Answer:** A. Dropping hidden units with a new mask each step corrupts the memory; masking U once per pass leaves the recurrence intact and keeps cuDNN usable — A mask on $U$ drawn once per forward pass is a fixed perturbation of the recurrent dynamics for that batch, not noise injected into $h_t$ at every step. The LSTM itself is unchanged, so the fused kernel can run it. AWD-LSTM still uses variational and embedding dropout elsewhere.

**Key points**

- DropConnect masks weights: $u=(M\odot W)v$, $M_{ij}\sim\mathrm{Bernoulli}(q)$; dropout masks units (whole columns of $W$).
- $\mathbb E[u]=qWv$ and $\operatorname{Var}[u]=q(1-q)(W\odot W)(v\odot v)$; for one unit the distribution is the same as dropout on its inputs.
- Mean inference $a(qWv)$ is biased after a nonlinearity; Wan et al. fit a Gaussian to $u$ and average $a(u)$ over samples.
- Per-example masks are costly; practical versions share one mask per batch.
- AWD-LSTM's weight-dropped LSTM masks the recurrent matrices once per forward pass.

---

From MLFoundry (https://www.mlfoundry.in). The web page adds interactive figures and diagrams.
