Definition
Tacotron 2 is a two-part text-to-speech system. A sequence-to-sequence spectrogram prediction network maps characters to an 80-band mel spectrogram, and a modified WaveNet vocoder turns that spectrogram into a waveform. It was introduced by Shen, Pang, Weiss, Schuster, Jaitly, Yang, Chen, Zhang, Wang, Skerry-Ryan, Saurous, Agiomyrgiannakis and Wu in "Natural TTS Synthesis by Conditioning WaveNet on Mel Spectrogram Predictions" (ICASSP 2018).
Tacotron had shown that one network could map characters to spectrograms. However, its Griffin–Lim phase reconstruction sounded metallic, and its content-only attention could lose its place. Separately, WaveNet produced very natural audio, but in earlier systems it was conditioned on linguistic features, durations and F0 from a hand-built front end.
Tacotron 2 joins the two through a simple interface: the mel spectrogram. It is a compact (80 numbers per 12.5 ms frame), smooth target that is easy to predict with a squared loss. It still carries enough detail for a neural vocoder to fill in phase and fine structure. The general two-stage split is covered in Text-to-speech.
Architecture
The encoder embeds each character as a 512-d vector. Three convolution layers (512 filters of width 5, Batch normalization, ReLU) model local context, like n-grams. A single Bidirectional LSTM with 256 units each way then gives one 512-d state per character. Compared with Tacotron's CBHG and GRUs, these are plain building blocks.
Location-sensitive attention
Content-based attention scores each character only by how well it matches the decoder state. If a word contains the same letter twice, or a phrase repeats, the scores cannot tell the copies apart. Tacotron 2 uses location-sensitive attention (Chorowski et al., 2015), which adds features computed from where the attention has already been.
- decoder LSTM state at step (the query)
- encoder state of character
- 32 learned 1-D convolution filters of length 31, applied along the character axis to the cumulative attention weights
- 32-d location feature at character . , and are projected to 128-d before the tanh
- alignment over characters; context
The cumulative weights record how much each character has already been read. A filter can learn that the next character to read is just past the mass already consumed. This pushes the attention to move forward and makes repeating or skipping much less likely. Common open-source versions feed both the previous and the cumulative weights to the convolution.
Decoder, post-net and stop token
The previous mel frame passes through a pre-net of two 256-unit ReLU layers with dropout 0.5. The dropout is also kept on at inference, which adds output variation. The pre-net output and the context vector feed two uni-directional Long short-term memory (LSTM) layers of 1024 units, regularised with zoneout 0.1. A linear projection of the LSTM output and the context gives the next frame. There is no reduction factor: one frame is produced per step. Training uses Teacher forcing, so the true previous frame is fed in.
- mel frames before and after the post-net. The post-net has 5 convolutions with 512 filters of width 5, batch norm, and tanh on all but the last layer
- summed mean squared error before and after the post-net (plus the stop-token loss)
- probability that the utterance has ended. Generation stops at the first step with , rather than after a fixed number of steps
The vocoder is a WaveNet with 30 dilated causal convolution layers in 3 cycles. The dilation of layer is , i.e. repeated three times. The mel frames are upsampled to the sample rate by 2 upsampling layers instead of 3. Instead of a 256-way softmax over μ-law values, it outputs a 10-component mixture of logistic distributions, which gives 16-bit samples at 24 kHz. It is trained on mel spectrograms that the prediction network produces with teacher forcing, so it learns to work with the predictor's slightly imperfect output.
| System | MOS |
|---|---|
| Parametric (production) | 3.492 ± 0.096 |
| Tacotron (Griffin–Lim) | 4.001 ± 0.087 |
| Concatenative (production) | 4.166 ± 0.091 |
| WaveNet (linguistic features) | 4.341 ± 0.051 |
| Tacotron 2 | 4.526 ± 0.066 |
| Ground-truth recordings | 4.582 ± 0.053 |
Setup (Shen et al., 2018)
- Data
- internal US English corpus, 24.6 hours from one professional female speaker
- Mel features
- 50 ms Hann window, 12.5 ms hop, 80 mel channels from 125 Hz to 7.6 kHz, log of the magnitude clipped below at 0.01
- Training
- prediction network: teacher forcing, batch 64, Adam, learning rate 10⁻³ decaying to 10⁻⁵; WaveNet trained afterwards on its predictions
- Vocoder
- 30 layers, 3 dilation cycles, receptive field 6,139 samples ≈ 255.8 ms at 24 kHz
Content-only vs location-sensitive, live
Worked example: one attention step
| Character j | α_prev | f_j | Content e | α, content only | e with location | α, location-sensitive |
|---|---|---|---|---|---|---|
| b | 0 | 0 | 0.5 | 0.0055 | 0.5 | 0.0005 |
| a (first) | 0.05 | 0.05 | 5 | 0.4945 | 5.3 | 0.0555 |
| n | 0.9 | 0.925 | 0.5 | 0.0055 | 6.05 | 0.1176 |
| a (second) | 0.05 | 0.5 | 5 | 0.4945 | 8.0 | 0.8264 |
Content alone splits the weight 0.4945 / 0.4945 between the two a's, so the context vector is an average of the wrong letter and the right one. The location term adds to the second a, which is one step ahead of the current position, and only to the first a. After the softmax the right a gets 0.826. The n keeps 0.118 because the filter also rewards staying put. A learned model balances staying and advancing using the content scores.
1import torch, torch.nn as nn, torch.nn.functional as F23class LocationSensitiveAttention(nn.Module):4 def __init__(self, query_dim=1024, memory_dim=512, attn_dim=128, n_filters=32, kernel=31):5 super().__init__()6 self.W = nn.Linear(query_dim, attn_dim, bias=False)7 self.V = nn.Linear(memory_dim, attn_dim, bias=True) # bias b inside the tanh8 self.U = nn.Linear(n_filters, attn_dim, bias=False)9 self.w = nn.Linear(attn_dim, 1, bias=False)10 # channels: previous and cumulative weights (open-source practice; the paper names cumulative)11 self.loc = nn.Conv1d(2, n_filters, kernel, padding=kernel // 2, bias=False)1213 def forward(self, s, h, prev, cum):14 # s: (B, 1024) decoder state, h: (B, N, 512) encoder states, prev/cum: (B, N)15 f = self.loc(torch.stack([prev, cum], dim=1)).transpose(1, 2) # (B, N, 32)16 e = self.w(torch.tanh(self.W(s)[:, None] + self.V(h) + self.U(f))).squeeze(-1)17 alpha = F.softmax(e, dim=-1) # (B, N)18 context = torch.bmm(alpha[:, None], h).squeeze(1) # (B, 512)19 return context, alpha, cum + alpha2021att = LocationSensitiveAttention()22B, N = 2, 4023h = torch.randn(B, N, 512)24prev = torch.zeros(B, N); prev[:, 0] = 1.0 # start on character 025c, a, cum = att(torch.randn(B, 1024), h, prev, prev.clone())26print(c.shape, a.shape) # torch.Size([2, 512]) torch.Size([2, 40])Use when
- +Near-human naturalness: MOS 4.53 vs 4.58 for recordings in the paper's test
- +Simple, modular blocks; the mel interface lets the vocoder be swapped (WaveRNN, WaveGlow, HiFi-GAN)
- +Location-sensitive attention and a stop token make alignment and ending far more reliable than in Tacotron
Avoid when
- −Autoregressive at two levels: about 80 decoder steps per second of audio, and the original WaveNet one sample at a time
- −Soft attention can still skip or repeat on hard inputs
- −Little direct control over speed or prosody
Check yourself
Content-based attention gives two identical letters exactly the same score. What lets location-sensitive attention pick the right one?
Check yourself
Tacotron 2's WaveNet receptive field is 6,139 samples at 24 kHz. Its mel frames have a 12.5 ms hop. About how many mel frames does one receptive field span?
Remember
- ✓Encoder: 512-d character embedding → 3 conv layers (512 × width 5) → BiLSTM (256 + 256).
- ✓Location-sensitive attention: with from 32 filters of length 31 over the cumulative attention weights.
- ✓Decoder: dropout pre-net (kept on at inference) → 2 LSTMs of 1024 → one 80-band mel frame per step (r = 1), plus a stop token at p > 0.5.
- ✓5-conv residual post-net; MSE before and after it. Mel: 50 ms window, 12.5 ms hop, 125 Hz–7.6 kHz.
- ✓Modified WaveNet (30 layers, 3 cycles, 10-component mixture of logistics, 24 kHz): MOS 4.526 vs 4.582 for ground truth.