# Conformer

> A speech encoder block that puts self-attention (global, by content) and a depthwise convolution module (local) between two half-step feed-forward layers. Conformer-L reached 1.9/3.9% LibriSpeech WER with an LM.

Source: https://www.mlfoundry.in/learn/conformer  
Section: Deep Learning › Speech and Audio Deep Learning  
Level: advanced · 8 min read · updated 2026-10-15

**Conformer:** (convolution-augmented Transformer) is the encoder architecture of Gulati, Qin, Chiu, Parmar, Zhang, Yu, Han, Wang, Zhang, Wu and Pang, "Conformer: Convolution-augmented Transformer for Speech Recognition" (Google, Interspeech 2020). Each block combines **multi-head self-attention** with a **convolution module**. Two half-step feed-forward layers surround them, in a "macaron" layout. Stacks of these blocks became the default acoustic encoder for CTC, transducer and attention-based recognisers.

An ASR encoder turns a sequence of filterbank frames into vectors that a decoder can read off as text. Two kinds of structure matter. The local kind spans tens of milliseconds: formant transitions, the burst of a "t", coarticulation between neighbouring phones. The global kind spans the utterance: speaker, channel, and words that repeat.

Convolutions are good at the local part and cheap, but a stack of them sees only a bounded window. Self-attention (see **Self-attention**) links any two frames in one step, by content. However, it has no built-in sense of "next to". Before Conformer, the strongest LibriSpeech encoders were either convolutional (ContextNet) or Transformer-based (Transformer transducers). Conformer puts both in every block.

## The block

$$
\tilde x=x+\tfrac12\,\mathrm{FFN}(x),\quad x'=\tilde x+\mathrm{MHSA}(\tilde x),\quad x''=x'+\mathrm{Conv}(x'),\quad y=\mathrm{LayerNorm}\big(x''+\tfrac12\,\mathrm{FFN}(x'')\big)
$$

- $\mathrm{FFN}$: LayerNorm → Linear $d\to4d$ → Swish → dropout → Linear $4d\to d$ → dropout. Two of them, each added with weight ½
- $\mathrm{MHSA}$: LayerNorm → multi-head self-attention with **relative** sinusoidal positional encoding (the Transformer-XL scheme) → dropout
- $\mathrm{Conv}$: the convolution module: LayerNorm → pointwise conv $d\to2d$ → GLU → depthwise conv → BatchNorm → Swish → pointwise conv → dropout

> Figure: Left: the encoder pipeline, which goes from 80-bin log-mel filterbanks through SpecAugment and convolutional subsampling to a linear projection, N Conformer blocks and an RNN-T decoder. Middle: one block. A half-step feed-forward, relative-position MHSA, the convolution module and a second half-step feed-forward are each added residually, and a final LayerNorm closes the block. Right: the convolution module, where a GLU gate halves the doubled width and a depthwise convolution with kernel 32 mixes neighbouring frames. Bottom: the S, M and L configurations.

The **macaron** layout comes from Macaron-Net (Lu et al., 2019), which reads a Transformer block as a numerical ODE solver. Splitting the feed-forward into two half steps, one before attention and one after convolution, works better than a single full FFN. In the Conformer ablations, replacing the macaron pair with one FFN raised WER.

$$
\mathrm{GLU}(a,b)=a\odot\sigma(b),\qquad z_{t,c}=\sum_{k=0}^{K-1}w_{k,c}\,u_{t+k-\lfloor K/2\rfloor,\,c}
$$

- $a,\,b$: the two halves of the pointwise conv's $2d$ output channels. The sigmoid half gates the other, giving $d$ channels again
- $w_{k,c}$: depthwise kernel: one $K$-tap filter **per channel** $c$, with no mixing across channels. See **Depthwise separable convolution**
- $K$: kernel size in frames; 32 in the paper, chosen by ablation over 3, 7, 17, 32 and 65

The front end is standard. It takes 80-channel filterbanks with 25 ms windows and a 10 ms hop, applies SpecAugment masking during training, and then a convolutional subsampling layer. Common implementations subsample 4× in time, to 40 ms frames. The paper decodes with a single-layer LSTM transducer (RNN-T); the same encoder is widely used with CTC or attention decoders. Self-attention uses pre-norm residual units, and its relative positional encoding helps the model handle utterances of different lengths.

Conformer sizes and LibriSpeech WER (%) from the paper (test-clean / test-other)

| Model | Params | Blocks | Width d | Heads | Decoder LSTM | No LM | With LM |
| --- | --- | --- | --- | --- | --- | --- | --- |
| Conformer-S | 10.3M | 16 | 144 | 4 | 320 | 2.7 / 6.3 | 2.1 / 5.0 |
| Conformer-M | 30.7M | 16 | 256 | 4 | 640 | 2.3 / 5.0 | 2.0 / 4.3 |
| Conformer-L | 118.8M | 17 | 512 | 8 | 640 | 2.1 / 4.3 | 1.9 / 3.9 |

## Local plus global: how far each part sees

A stack of $L$ convolutions with kernel $K$ can reach at most $L(K-1)+1$ frames. That is only an upper bound: the **Receptive field** that actually carries weight grows much more slowly, roughly like $\sqrt L$, because paths through the middle of the kernels dominate. One attention layer reaches every frame, but it weights frames by content similarity, not by distance.

$$
\mathrm{RF}_{\text{conv}}(L,K)=L\,(K-1)+1\ \text{frames},\qquad \mathrm{RF}_{\text{attn}}=T\ \text{frames from the first layer}
$$

- $L,\,K$: number of stacked convolutions and their kernel size
- $T$: sequence length after subsampling: 30 s at 40 ms per frame is 750 frames

> Figure: 48 frames of 40 ms with the sounds s a t · m a p · s a t · t a p. Each bar shows how much an input frame feeds output frame 30 (an "a") through residual blocks of the form (I + mixing)/2. A fixed Hann-shaped kernel stands in for the learned depthwise filter. Attention is a softmax over cosine similarity of synthetic frame features, with no position term. Default (K = 15, L = 1): convolution puts 24.1% on neighbouring frames of other sounds and 0.0% on the distant a's, and 90% of its weight lies within ±4 frames (160 ms). Attention is the mirror image, 0.0% and 23.9%. The Conformer block gets both, 18.7% and 19.0%. At L = 4 the conv stack's reach is 57 frames, but 90% of its weight still sits within ±7, while attention's distant same-sound share rises to 59.2%.

**Worked example: Conformer-L numbers (d = 512, 17 blocks, K = 32, 40 ms frames)**

1. One conv module spans $K=32$ frames $=1.28$ s. All 17 stacked span at most $17\cdot31+1=528$ frames $=21.12$ s.
2. Conv module weights: pointwise $512\times1024=524{,}288$, depthwise $512\times32=16{,}384$, pointwise $512\times512=262{,}144$, total $802{,}816$.
3. MHSA projections: $4\times512^2=1{,}048{,}576$. Each FFN: $2\times512\times2048=2{,}097{,}152$, and there are two per block.
4. Per block, without biases, norms or the relative-position projection: $802{,}816+1{,}048{,}576+2\times2{,}097{,}152=6{,}045{,}696$. Over 17 blocks that is about $102.8$M of the $118.8$M total.
5. Attention cost on a 30 s utterance: $T=750$ frames, so $750^2=562{,}500$ scores per head per layer. The depthwise conv needs only $750\times32$ multiplies per channel.

```python
import torch, torchaudio
from torchaudio.models import Conformer

fbank = torchaudio.transforms.MelSpectrogram(sample_rate=16000, n_fft=400, hop_length=160, n_mels=80)
wave = torch.randn(1, 4 * 16000)                         # stand-in for 4 s of speech
x = (fbank(wave) + 1e-6).log()                           # (1, 80, 401): 100 frames/s

stem = torch.nn.Sequential(                              # 4x subsampling -> 40 ms frames
    torch.nn.Conv1d(80, 144, 3, stride=2, padding=1), torch.nn.SiLU(),
    torch.nn.Conv1d(144, 144, 3, stride=2, padding=1))
h = stem(x).transpose(1, 2)                              # (1, 101, 144)

enc = Conformer(input_dim=144, num_heads=4, ffn_dim=576, # Conformer-S width, FFN = 4d
                num_layers=16, depthwise_conv_kernel_size=31)  # torchaudio needs an odd kernel
y, lengths = enc(h, torch.tensor([h.shape[1]]))
print(y.shape)                                           # torch.Size([1, 101, 144])
```

A Conformer encoder in torchaudio on 4 s of audio. torchaudio's layer uses absolute (not relative) attention and leaves subsampling to you, so a 4× conv stem is added by hand (not run here: needs torch)

> **What the ablations said:** In the paper, removing the convolution module hurt more than any other single change, especially on test-other. Placing convolution **after** self-attention worked better than before it. Splitting the FFN into macaron halves and using Swish rather than ReLU each gave smaller gains. Attention and convolution are complementary, not redundant.

> **BatchNorm inside the conv module:** The conv module uses BatchNorm, which depends on batch statistics. With very small or highly variable batches, or in streaming inference, the train and test behaviour of BatchNorm can drift apart. Many implementations swap it for LayerNorm or GroupNorm; torchaudio exposes this as `use_group_norm`. Streaming models must also make the depthwise conv **causal** (left context only) and limit attention to past frames.

- **Transducer and CTC ASR:** Conformer-Transducer and Conformer-CTC models in open toolkits such as ESPnet and NVIDIA NeMo
- **Self-supervised speech:** Conformer encoders pre-trained with masked objectives, e.g. w2v-BERT and Google's Universal Speech Model, before fine-tuning
- **Beyond ASR:** speech translation, keyword spotting, speaker and emotion recognition: any task needing local acoustic detail plus utterance context

## Descendants

Later encoders kept the idea and reshuffled the parts. Branchformer and E-Branchformer (2022) run attention and convolution in parallel branches and merge them. Squeezeformer (2022) changes the block order and adds temporal down- and up-sampling inside the stack. Zipformer (2023) runs different stacks at different frame rates. All of them still pair a global, content-based mixer with a local convolutional one.

**Strengths**

- Local and global context in every block; state-of-the-art LibriSpeech WER at each model size when published
- Depthwise convolution adds locality for few parameters (16,384 depthwise weights per block at d = 512)
- Works with CTC, transducer and attention decoders, and as a self-supervised encoder

**Limitations**

- Attention is still O(T²) in sequence length; long audio needs subsampling or chunking
- BatchNorm and non-causal convolution complicate streaming
- More moving parts than a plain Transformer; implementations differ (positions, norms, kernel size)

**Question:** Why does a Conformer block include a convolution module if self-attention can already reach every frame?

- A. Self-attention cannot process sequences longer than 32 frames
- B. Attention weights frames by content similarity and has no built-in locality; the depthwise convolution adds cheap, position-aware mixing of neighbouring frames
- C. The convolution replaces the positional encoding entirely
- D. Convolutions are needed to compute the softmax

**Answer:** B. Attention weights frames by content similarity and has no built-in locality; the depthwise convolution adds cheap, position-aware mixing of neighbouring frames — In the interactive figure, attention-only mixing puts 0.0% on neighbouring frames of other sounds: it attends to similar frames wherever they are. The conv module supplies exactly that local context, at $O(TK)$ cost instead of $O(T^2)$.

**Question:** Six convolutions of kernel 32 are stacked over 40 ms frames. What is the maximum span of one output frame's input?

- A. 192 frames (7.68 s)
- B. 187 frames (7.48 s)
- C. 32 frames (1.28 s)
- D. Unlimited

**Answer:** B. 187 frames (7.48 s) — $L(K-1)+1=6\cdot31+1=187$ frames $=7.48$ s. The weight that actually matters is concentrated in a much narrower centre, which is one reason attention is still useful.

**Key points**

- Block: $x+\tfrac12\mathrm{FFN}$ → $+\mathrm{MHSA}$ (relative positions) → $+\mathrm{Conv}$ → $+\tfrac12\mathrm{FFN}$ → LayerNorm.
- Conv module: LayerNorm → pointwise d→2d → GLU → depthwise (K = 32) → BatchNorm → Swish → pointwise → dropout.
- Attention mixes globally by content; convolution mixes locally by position; Conformer uses both in every block.
- Conv stack reach $L(K-1)+1$ frames is only an upper bound; effective reach grows far more slowly.
- Conformer-L (118.8M): 2.1/4.3% LibriSpeech WER without an LM, 1.9/3.9% with one.

---

From MLFoundry (https://www.mlfoundry.in). The web page adds interactive figures and diagrams.
