Definition
Conformer (convolution-augmented Transformer) is the encoder architecture of Gulati, Qin, Chiu, Parmar, Zhang, Yu, Han, Wang, Zhang, Wu and Pang, "Conformer: Convolution-augmented Transformer for Speech Recognition" (Google, Interspeech 2020). Each block combines multi-head self-attention with a convolution module. Two half-step feed-forward layers surround them, in a "macaron" layout. Stacks of these blocks became the default acoustic encoder for CTC, transducer and attention-based recognisers.
An ASR encoder turns a sequence of filterbank frames into vectors that a decoder can read off as text. Two kinds of structure matter. The local kind spans tens of milliseconds: formant transitions, the burst of a "t", coarticulation between neighbouring phones. The global kind spans the utterance: speaker, channel, and words that repeat.
Convolutions are good at the local part and cheap, but a stack of them sees only a bounded window. Self-attention (see Self-attention) links any two frames in one step, by content. However, it has no built-in sense of "next to". Before Conformer, the strongest LibriSpeech encoders were either convolutional (ContextNet) or Transformer-based (Transformer transducers). Conformer puts both in every block.
The block
- LayerNorm → Linear → Swish → dropout → Linear → dropout. Two of them, each added with weight ½
- LayerNorm → multi-head self-attention with relative sinusoidal positional encoding (the Transformer-XL scheme) → dropout
- the convolution module: LayerNorm → pointwise conv → GLU → depthwise conv → BatchNorm → Swish → pointwise conv → dropout
The macaron layout comes from Macaron-Net (Lu et al., 2019), which reads a Transformer block as a numerical ODE solver. Splitting the feed-forward into two half steps, one before attention and one after convolution, works better than a single full FFN. In the Conformer ablations, replacing the macaron pair with one FFN raised WER.
- the two halves of the pointwise conv's output channels. The sigmoid half gates the other, giving channels again
- depthwise kernel: one -tap filter per channel , with no mixing across channels. See Depthwise separable convolution
- kernel size in frames; 32 in the paper, chosen by ablation over 3, 7, 17, 32 and 65
The front end is standard. It takes 80-channel filterbanks with 25 ms windows and a 10 ms hop, applies SpecAugment masking during training, and then a convolutional subsampling layer. Common implementations subsample 4× in time, to 40 ms frames. The paper decodes with a single-layer LSTM transducer (RNN-T); the same encoder is widely used with CTC or attention decoders. Self-attention uses pre-norm residual units, and its relative positional encoding helps the model handle utterances of different lengths.
| Model | Params | Blocks | Width d | Heads | Decoder LSTM | No LM | With LM |
|---|---|---|---|---|---|---|---|
| Conformer-S | 10.3M | 16 | 144 | 4 | 320 | 2.7 / 6.3 | 2.1 / 5.0 |
| Conformer-M | 30.7M | 16 | 256 | 4 | 640 | 2.3 / 5.0 | 2.0 / 4.3 |
| Conformer-L | 118.8M | 17 | 512 | 8 | 640 | 2.1 / 4.3 | 1.9 / 3.9 |
Local plus global: how far each part sees
A stack of convolutions with kernel can reach at most frames. That is only an upper bound: the Receptive field that actually carries weight grows much more slowly, roughly like , because paths through the middle of the kernels dominate. One attention layer reaches every frame, but it weights frames by content similarity, not by distance.
- number of stacked convolutions and their kernel size
- sequence length after subsampling: 30 s at 40 ms per frame is 750 frames
Worked example: Conformer-L numbers (d = 512, 17 blocks, K = 32, 40 ms frames)
- 1One conv module spans frames s. All 17 stacked span at most frames s.
- 2Conv module weights: pointwise , depthwise , pointwise , total .
- 3MHSA projections: . Each FFN: , and there are two per block.
- 4Per block, without biases, norms or the relative-position projection: . Over 17 blocks that is about M of the M total.
- 5Attention cost on a 30 s utterance: frames, so scores per head per layer. The depthwise conv needs only multiplies per channel.
1import torch, torchaudio2from torchaudio.models import Conformer34fbank = torchaudio.transforms.MelSpectrogram(sample_rate=16000, n_fft=400, hop_length=160, n_mels=80)5wave = torch.randn(1, 4 * 16000) # stand-in for 4 s of speech6x = (fbank(wave) + 1e-6).log() # (1, 80, 401): 100 frames/s78stem = torch.nn.Sequential( # 4x subsampling -> 40 ms frames9 torch.nn.Conv1d(80, 144, 3, stride=2, padding=1), torch.nn.SiLU(),10 torch.nn.Conv1d(144, 144, 3, stride=2, padding=1))11h = stem(x).transpose(1, 2) # (1, 101, 144)1213enc = Conformer(input_dim=144, num_heads=4, ffn_dim=576, # Conformer-S width, FFN = 4d14 num_layers=16, depthwise_conv_kernel_size=31) # torchaudio needs an odd kernel15y, lengths = enc(h, torch.tensor([h.shape[1]]))16print(y.shape) # torch.Size([1, 101, 144])Where it's used
Transducer and CTC ASR
Conformer-Transducer and Conformer-CTC models in open toolkits such as ESPnet and NVIDIA NeMo
Self-supervised speech
Conformer encoders pre-trained with masked objectives, e.g. w2v-BERT and Google's Universal Speech Model, before fine-tuning
Beyond ASR
speech translation, keyword spotting, speaker and emotion recognition: any task needing local acoustic detail plus utterance context
Descendants
Later encoders kept the idea and reshuffled the parts. Branchformer and E-Branchformer (2022) run attention and convolution in parallel branches and merge them. Squeezeformer (2022) changes the block order and adds temporal down- and up-sampling inside the stack. Zipformer (2023) runs different stacks at different frame rates. All of them still pair a global, content-based mixer with a local convolutional one.
Use when
- +Local and global context in every block; state-of-the-art LibriSpeech WER at each model size when published
- +Depthwise convolution adds locality for few parameters (16,384 depthwise weights per block at d = 512)
- +Works with CTC, transducer and attention decoders, and as a self-supervised encoder
Avoid when
- −Attention is still O(T²) in sequence length; long audio needs subsampling or chunking
- −BatchNorm and non-causal convolution complicate streaming
- −More moving parts than a plain Transformer; implementations differ (positions, norms, kernel size)
Check yourself
Why does a Conformer block include a convolution module if self-attention can already reach every frame?
Check yourself
Six convolutions of kernel 32 are stacked over 40 ms frames. What is the maximum span of one output frame's input?
Remember
- ✓Block: → (relative positions) → → → LayerNorm.
- ✓Conv module: LayerNorm → pointwise d→2d → GLU → depthwise (K = 32) → BatchNorm → Swish → pointwise → dropout.
- ✓Attention mixes globally by content; convolution mixes locally by position; Conformer uses both in every block.
- ✓Conv stack reach frames is only an upper bound; effective reach grows far more slowly.
- ✓Conformer-L (118.8M): 2.1/4.3% LibriSpeech WER without an LM, 1.9/3.9% with one.