Conformer

A speech encoder block that puts self-attention (global, by content) and a depthwise convolution module (local) between two half-step feed-forward layers. Conformer-L reached 1.9/3.9% LibriSpeech WER with an LM.

8 min read708 wordsadvanced0 viewsupdated 2026-10-15

Definition

Conformer (convolution-augmented Transformer) is the encoder architecture of Gulati, Qin, Chiu, Parmar, Zhang, Yu, Han, Wang, Zhang, Wu and Pang, "Conformer: Convolution-augmented Transformer for Speech Recognition" (Google, Interspeech 2020). Each block combines multi-head self-attention with a convolution module. Two half-step feed-forward layers surround them, in a "macaron" layout. Stacks of these blocks became the default acoustic encoder for CTC, transducer and attention-based recognisers.

An ASR encoder turns a sequence of filterbank frames into vectors that a decoder can read off as text. Two kinds of structure matter. The local kind spans tens of milliseconds: formant transitions, the burst of a "t", coarticulation between neighbouring phones. The global kind spans the utterance: speaker, channel, and words that repeat.

Convolutions are good at the local part and cheap, but a stack of them sees only a bounded window. Self-attention (see Self-attention) links any two frames in one step, by content. However, it has no built-in sense of "next to". Before Conformer, the strongest LibriSpeech encoders were either convolutional (ContextNet) or Transformer-based (Transformer transducers). Conformer puts both in every block.

The block

x~=x+12 FFN(x),x′=x~+MHSA(x~),x′′=x′+Conv(x′),y=LayerNorm(x′′+12 FFN(x′′))\tilde x=x+\tfrac12\,\mathrm{FFN}(x),\quad x'=\tilde x+\mathrm{MHSA}(\tilde x),\quad x''=x'+\mathrm{Conv}(x'),\quad y=\mathrm{LayerNorm}\big(x''+\tfrac12\,\mathrm{FFN}(x'')\big)
FFN\mathrm{FFN}
LayerNorm → Linear d→4dd\to4d → Swish → dropout → Linear 4d→d4d\to d → dropout. Two of them, each added with weight ½
MHSA\mathrm{MHSA}
LayerNorm → multi-head self-attention with relative sinusoidal positional encoding (the Transformer-XL scheme) → dropout
Conv\mathrm{Conv}
the convolution module: LayerNorm → pointwise conv d→2dd\to2d → GLU → depthwise conv → BatchNorm → Swish → pointwise conv → dropout
encoder pipelinelog-mel filterbank80 bins · 25 ms / 10 msSpecAugmenttime + frequency masksconv subsamplingfewer, longer frameslinear + dropoutproject to width dN × Conformer block(expanded on the right)RNN-T decoder1-layer LSTMone Conformer blockx (frames × d)feed-forwardd → 4d → d, Swish× ½+MHSArelative pos. enc.+convolutionmodule (right)+feed-forwardd → 4d → d, Swish× ½+LayerNormy (frames × d)convolution moduleLayerNormpointwise conv d → 2dGLU (gate) 2d → ddepthwise conv, K = 32BatchNormSwishpointwise conv d → ddropoutx̃ = x + ½FFN(x) · x′ = x̃ + MHSA(x̃) · x″ = x′ + Conv(x′) · y = LayerNorm(x″ + ½FFN(x″))each FFN, MHSA and conv branch starts with LayerNorm (pre-norm) and ends with dropoutS: 16 blocks, d 144, 4 heads, 10.3M · M: 16, d 256, 4 heads, 30.7M · L: 17, d 512, 8 heads, 118.8M
fig.Left: the encoder pipeline, which goes from 80-bin log-mel filterbanks through SpecAugment and convolutional subsampling to a linear projection, N Conformer blocks and an RNN-T decoder. Middle: one block. A half-step feed-forward, relative-position MHSA, the convolution module and a second half-step feed-forward are each added residually, and a final LayerNorm closes the block. Right: the convolution module, where a GLU gate halves the doubled width and a depthwise convolution with kernel 32 mixes neighbouring frames. Bottom: the S, M and L configurations.

The macaron layout comes from Macaron-Net (Lu et al., 2019), which reads a Transformer block as a numerical ODE solver. Splitting the feed-forward into two half steps, one before attention and one after convolution, works better than a single full FFN. In the Conformer ablations, replacing the macaron pair with one FFN raised WER.

GLU(a,b)=a⊙σ(b),zt,c=∑k=0K−1wk,c ut+k−⌊K/2⌋, c\mathrm{GLU}(a,b)=a\odot\sigma(b),\qquad z_{t,c}=\sum_{k=0}^{K-1}w_{k,c}\,u_{t+k-\lfloor K/2\rfloor,\,c}
a, ba,\,b
the two halves of the pointwise conv's 2d2d output channels. The sigmoid half gates the other, giving dd channels again
wk,cw_{k,c}
depthwise kernel: one KK-tap filter per channel cc, with no mixing across channels. See Depthwise separable convolution
KK
kernel size in frames; 32 in the paper, chosen by ablation over 3, 7, 17, 32 and 65

The front end is standard. It takes 80-channel filterbanks with 25 ms windows and a 10 ms hop, applies SpecAugment masking during training, and then a convolutional subsampling layer. Common implementations subsample 4× in time, to 40 ms frames. The paper decodes with a single-layer LSTM transducer (RNN-T); the same encoder is widely used with CTC or attention decoders. Self-attention uses pre-norm residual units, and its relative positional encoding helps the model handle utterances of different lengths.

ModelParamsBlocksWidth dHeadsDecoder LSTMNo LMWith LM
Conformer-S10.3M1614443202.7 / 6.32.1 / 5.0
Conformer-M30.7M1625646402.3 / 5.02.0 / 4.3
Conformer-L118.8M1751286402.1 / 4.31.9 / 3.9
Conformer sizes and LibriSpeech WER (%) from the paper (test-clean / test-other)

Local plus global: how far each part sees

A stack of LL convolutions with kernel KK can reach at most L(K−1)+1L(K-1)+1 frames. That is only an upper bound: the Receptive field that actually carries weight grows much more slowly, roughly like L\sqrt L, because paths through the middle of the kernels dominate. One attention layer reaches every frame, but it weights frames by content similarity, not by distance.

RFconv(L,K)=L (K−1)+1 frames,RFattn=T frames from the first layer\mathrm{RF}_{\text{conv}}(L,K)=L\,(K-1)+1\ \text{frames},\qquad \mathrm{RF}_{\text{attn}}=T\ \text{frames from the first layer}
L, KL,\,K
number of stacked convolutions and their kernel size
TT
sequence length after subsampling: 30 s at 40 ms per frame is 750 frames
influence of input frames on output frame 30 (“a”) after 1 block · 48 frames × 40 ms·sat·map·sat·tap·soundconvnear, other sound 24.1% · same sound, far 0.0% · 90% within ±4 fr (160 ms)self 56.3%attentionnear, other sound 0.0% · same sound, far 23.9% · 90% within ±21 fr (840 ms)self 62.4%conformernear, other sound 18.7% · same sound, far 19.0% · 90% within ±21 fr (840 ms)self 35.6%shaded ±5 frames (±200 ms) · solid: same sound · red: frame 30 itself (clipped)conv stack reach L(K−1)+1 = 1·14+1 = 15 frames (0.60 s) · attention: all 48 frames at once
fig.48 frames of 40 ms with the sounds s a t · m a p · s a t · t a p. Each bar shows how much an input frame feeds output frame 30 (an "a") through residual blocks of the form (I + mixing)/2. A fixed Hann-shaped kernel stands in for the learned depthwise filter. Attention is a softmax over cosine similarity of synthetic frame features, with no position term. Default (K = 15, L = 1): convolution puts 24.1% on neighbouring frames of other sounds and 0.0% on the distant a's, and 90% of its weight lies within ±4 frames (160 ms). Attention is the mirror image, 0.0% and 23.9%. The Conformer block gets both, 18.7% and 19.0%. At L = 4 the conv stack's reach is 57 frames, but 90% of its weight still sits within ±7, while attention's distant same-sound share rises to 59.2%.

Worked example: Conformer-L numbers (d = 512, 17 blocks, K = 32, 40 ms frames)

  1. 1One conv module spans K=32K=32 frames =1.28=1.28 s. All 17 stacked span at most 17⋅31+1=52817\cdot31+1=528 frames =21.12=21.12 s.
  2. 2Conv module weights: pointwise 512×1024=524,288512\times1024=524{,}288, depthwise 512×32=16,384512\times32=16{,}384, pointwise 512×512=262,144512\times512=262{,}144, total 802,816802{,}816.
  3. 3MHSA projections: 4×5122=1,048,5764\times512^2=1{,}048{,}576. Each FFN: 2×512×2048=2,097,1522\times512\times2048=2{,}097{,}152, and there are two per block.
  4. 4Per block, without biases, norms or the relative-position projection: 802,816+1,048,576+2×2,097,152=6,045,696802{,}816+1{,}048{,}576+2\times2{,}097{,}152=6{,}045{,}696. Over 17 blocks that is about 102.8102.8M of the 118.8118.8M total.
  5. 5Attention cost on a 30 s utterance: T=750T=750 frames, so 7502=562,500750^2=562{,}500 scores per head per layer. The depthwise conv needs only 750×32750\times32 multiplies per channel.
python
1import torch, torchaudio
2from torchaudio.models import Conformer
3
4fbank = torchaudio.transforms.MelSpectrogram(sample_rate=16000, n_fft=400, hop_length=160, n_mels=80)
5wave = torch.randn(1, 4 * 16000) # stand-in for 4 s of speech
6x = (fbank(wave) + 1e-6).log() # (1, 80, 401): 100 frames/s
7
8stem = torch.nn.Sequential( # 4x subsampling -> 40 ms frames
9 torch.nn.Conv1d(80, 144, 3, stride=2, padding=1), torch.nn.SiLU(),
10 torch.nn.Conv1d(144, 144, 3, stride=2, padding=1))
11h = stem(x).transpose(1, 2) # (1, 101, 144)
12
13enc = Conformer(input_dim=144, num_heads=4, ffn_dim=576, # Conformer-S width, FFN = 4d
14 num_layers=16, depthwise_conv_kernel_size=31) # torchaudio needs an odd kernel
15y, lengths = enc(h, torch.tensor([h.shape[1]]))
16print(y.shape) # torch.Size([1, 101, 144])
A Conformer encoder in torchaudio on 4 s of audio. torchaudio's layer uses absolute (not relative) attention and leaves subsampling to you, so a 4× conv stem is added by hand (not run here: needs torch)

Where it's used

Transducer and CTC ASR

Conformer-Transducer and Conformer-CTC models in open toolkits such as ESPnet and NVIDIA NeMo

Self-supervised speech

Conformer encoders pre-trained with masked objectives, e.g. w2v-BERT and Google's Universal Speech Model, before fine-tuning

Beyond ASR

speech translation, keyword spotting, speaker and emotion recognition: any task needing local acoustic detail plus utterance context

Descendants

Later encoders kept the idea and reshuffled the parts. Branchformer and E-Branchformer (2022) run attention and convolution in parallel branches and merge them. Squeezeformer (2022) changes the block order and adds temporal down- and up-sampling inside the stack. Zipformer (2023) runs different stacks at different frame rates. All of them still pair a global, content-based mixer with a local convolutional one.

Use when

  • +Local and global context in every block; state-of-the-art LibriSpeech WER at each model size when published
  • +Depthwise convolution adds locality for few parameters (16,384 depthwise weights per block at d = 512)
  • +Works with CTC, transducer and attention decoders, and as a self-supervised encoder

Avoid when

  • −Attention is still O(T²) in sequence length; long audio needs subsampling or chunking
  • −BatchNorm and non-causal convolution complicate streaming
  • −More moving parts than a plain Transformer; implementations differ (positions, norms, kernel size)

Check yourself

Why does a Conformer block include a convolution module if self-attention can already reach every frame?

Check yourself

Six convolutions of kernel 32 are stacked over 40 ms frames. What is the maximum span of one output frame's input?

Remember

  • ✓Block: x+12FFNx+\tfrac12\mathrm{FFN} → +MHSA+\mathrm{MHSA} (relative positions) → +Conv+\mathrm{Conv} → +12FFN+\tfrac12\mathrm{FFN} → LayerNorm.
  • ✓Conv module: LayerNorm → pointwise d→2d → GLU → depthwise (K = 32) → BatchNorm → Swish → pointwise → dropout.
  • ✓Attention mixes globally by content; convolution mixes locally by position; Conformer uses both in every block.
  • ✓Conv stack reach L(K−1)+1L(K-1)+1 frames is only an upper bound; effective reach grows far more slowly.
  • ✓Conformer-L (118.8M): 2.1/4.3% LibriSpeech WER without an LM, 1.9/3.9% with one.
← back to Deep Learning roadmap