Tacotron 2

Shen et al. (2018): a conv + BiLSTM encoder, location-sensitive attention and an LSTM decoder predict one 80-band mel frame per step plus a stop token; a modified WaveNet turns it into audio. MOS 4.53 vs 4.58 for recordings.

8 min read794 wordscore0 viewsupdated 2026-10-15

Definition

Tacotron 2 is a two-part text-to-speech system. A sequence-to-sequence spectrogram prediction network maps characters to an 80-band mel spectrogram, and a modified WaveNet vocoder turns that spectrogram into a waveform. It was introduced by Shen, Pang, Weiss, Schuster, Jaitly, Yang, Chen, Zhang, Wang, Skerry-Ryan, Saurous, Agiomyrgiannakis and Wu in "Natural TTS Synthesis by Conditioning WaveNet on Mel Spectrogram Predictions" (ICASSP 2018).

Tacotron had shown that one network could map characters to spectrograms. However, its Griffin–Lim phase reconstruction sounded metallic, and its content-only attention could lose its place. Separately, WaveNet produced very natural audio, but in earlier systems it was conditioned on linguistic features, durations and F0 from a hand-built front end.

Tacotron 2 joins the two through a simple interface: the mel spectrogram. It is a compact (80 numbers per 12.5 ms frame), smooth target that is easy to predict with a squared loss. It still carries enough detail for a neural vocoder to fill in phase and fine structure. The general two-stage split is covered in Text-to-speech.

Architecture

The encoder embeds each character as a 512-d vector. Three convolution layers (512 filters of width 5, Batch normalization, ReLU) model local context, like n-grams. A single Bidirectional LSTM with 256 units each way then gives one 512-d state hjh_j per character. Compared with Tacotron's CBHG and GRUs, these are plain building blocks.

Tacotron 2 (Shen et al., 2018) · spectrogram prediction network + modified WaveNet vocodercharacters“banana…”→ 512-dembedding3 conv layers512 filters, width 5batch norm, ReLUdropout 0.5BiLSTM256 + 256 units→ h_j, 512-dper characterencoder stateslocation-sensitive attentione_ij = wᵀ tanh(W s_i + V h_j + U f_ij + b)f_i = F ∗ cumulative α (32 filters, length 31)W s, V h, U f projected to 128-dα_i = softmax(e_i), c_i = Σ_j α_ij h_jlearned alignment →pre-netprevious frame2 × FC-256, ReLUdropout 0.5, kepton at inference(<GO> frame first)decoder2 LSTMs, 1024zoneout 0.1input: pre-net⊕ context c_ir = 1 frame/steps_i ⇄ c_ilinear → 80-d mel framefed back to the pre-netstop tokenlinear → sigmoidstop when p > 0.5post-net: 5 conv, 512 × width 5BN, tanh (not last) · residual +mel 80 × Tmodified WaveNet30 dilated conv layers3 cycles: 1, 2, …, 5122 upsampling layers10-way mix. of logistics24 kHz, 16-bit waveformmel: 50 ms Hann window, 12.5 ms hop, 125 Hz–7.6 kHz · loss: MSE before + after post-net
fig.Tacotron 2 as published. The encoder has 3 convolutions and a BiLSTM. Location-sensitive attention adds features computed by 32 filters of length 31 over the cumulative attention weights. The decoder has a pre-net (2 × 256 units, with dropout kept on at inference) and 2 LSTMs of 1024 units, and emits one 80-band mel frame and one stop probability per step. A 5-layer convolutional post-net adds a residual correction, and a modified WaveNet (30 dilated layers in 3 cycles, a 10-component mixture of logistics) produces 24 kHz, 16-bit audio.

Location-sensitive attention

Content-based attention scores each character only by how well it matches the decoder state. If a word contains the same letter twice, or a phrase repeats, the scores cannot tell the copies apart. Tacotron 2 uses location-sensitive attention (Chorowski et al., 2015), which adds features computed from where the attention has already been.

ei,j=w⊤tanh⁡ ⁣(Wsi+Vhj+Ufi,j+b),fi=F∗∑k<iαk,αi=softmax(ei)e_{i,j}=w^{\top}\tanh\!\big(W s_i+V h_j+U f_{i,j}+b\big),\qquad f_i=F*\textstyle\sum_{k<i}\alpha_k,\qquad \alpha_i=\mathrm{softmax}(e_i)
sis_i
decoder LSTM state at step ii (the query)
hjh_j
encoder state of character jj
FF
32 learned 1-D convolution filters of length 31, applied along the character axis to the cumulative attention weights
fi,jf_{i,j}
32-d location feature at character jj. WsW s, VhV h and UfU f are projected to 128-d before the tanh
αi\alpha_i
alignment over characters; context ci=∑jαi,jhjc_i=\sum_j\alpha_{i,j}h_j

The cumulative weights ∑k<iαk\sum_{k<i}\alpha_k record how much each character has already been read. A filter can learn that the next character to read is just past the mass already consumed. This pushes the attention to move forward and makes repeating or skipping much less likely. Common open-source versions feed both the previous and the cumulative weights to the convolution.

Decoder, post-net and stop token

The previous mel frame passes through a pre-net of two 256-unit ReLU layers with dropout 0.5. The dropout is also kept on at inference, which adds output variation. The pre-net output and the context vector feed two uni-directional Long short-term memory (LSTM) layers of 1024 units, regularised with zoneout 0.1. A linear projection of the LSTM output and the context gives the next frame. There is no reduction factor: one frame is produced per step. Training uses Teacher forcing, so the true previous frame is fed in.

y^post=y^+Postnet(y^),L=∥y−y^∥22+∥y−y^post∥22,pistop=σ(u⊤[si;ci]+bs)\hat y_{\text{post}}=\hat y+\mathrm{Postnet}(\hat y),\qquad \mathcal L=\lVert y-\hat y\rVert_2^2+\lVert y-\hat y_{\text{post}}\rVert_2^2,\qquad p^{\text{stop}}_i=\sigma\big(u^{\top}[s_i;c_i]+b_s\big)
y^, y^post\hat y,\ \hat y_{\text{post}}
mel frames before and after the post-net. The post-net has 5 convolutions with 512 filters of width 5, batch norm, and tanh on all but the last layer
L\mathcal L
summed mean squared error before and after the post-net (plus the stop-token loss)
pistopp^{\text{stop}}_i
probability that the utterance has ended. Generation stops at the first step with p>0.5p>0.5, rather than after a fixed number of steps

The vocoder is a WaveNet with 30 dilated causal convolution layers in 3 cycles. The dilation of layer kk is 2k mod 102^{k\bmod10}, i.e. 1,2,…,5121,2,\dots,512 repeated three times. The mel frames are upsampled to the sample rate by 2 upsampling layers instead of 3. Instead of a 256-way softmax over μ-law values, it outputs a 10-component mixture of logistic distributions, which gives 16-bit samples at 24 kHz. It is trained on mel spectrograms that the prediction network produces with teacher forcing, so it learns to work with the predictor's slightly imperfect output.

SystemMOS
Parametric (production)3.492 ± 0.096
Tacotron (Griffin–Lim)4.001 ± 0.087
Concatenative (production)4.166 ± 0.091
WaveNet (linguistic features)4.341 ± 0.051
Tacotron 24.526 ± 0.066
Ground-truth recordings4.582 ± 0.053
Mean opinion scores on the paper's internal US English test set, 5-point scale (Shen et al., 2018)

Setup (Shen et al., 2018)

Data
internal US English corpus, 24.6 hours from one professional female speaker
Mel features
50 ms Hann window, 12.5 ms hop, 80 mel channels from 125 Hz to 7.6 kHz, log of the magnitude clipped below at 0.01
Training
prediction network: teacher forcing, batch 64, Adam, learning rate 10⁻³ decaying to 10⁻⁵; WaveNet trained afterwards on its predictions
Vocoder
30 layers, 3 dilation cycles, receptive field 6,139 samples ≈ 255.8 ms at 24 kHz

Content-only vs location-sensitive, live

alignment α(frame, character) · 70 frames · location-sensitive, λ = 6 · query noise σ = 0.30banana␣bandanaargmax on the spoken character: 68/70 frames · backward jumps 0 · characters never attended 0outlined: true alignment · red dots: argmax on a wrong character · mean max α = 0.94frame 24 (spoken: “a”, character 5) · e = β·⟨q, k⟩ + λ·(F ∗ α_prev), F = [stay 1, advance 0.5]banana␣bandanaeα■ content energy · ■ location energy · ■ α after softmax · α on spoken character = 0.78identical letters share a key here, so content energy alone cannot tell the six a’s apart
attention
fig.A toy decoder runs attention frame by frame over "banana bandana" (14 characters, 70 frames). Identical letters share one key, the hardest case. The location term is a fixed filter over the previous alignment that weights staying by 1 and moving one ahead by 0.5. At the default λ = 6 and σ = 0.3 the argmax is on the spoken character in 68 of 70 frames, with 0 backward jumps and no character skipped (mean peak 0.94). Press content only: 17 of 70 frames are correct, there are 9 backward jumps and 9 of the 14 characters are never attended. At frame 24, the first frame of the third letter, content-only attention spreads 0.16 over each of the six a's, while location-sensitive attention puts 0.78 on the right one. At σ = 0.6, λ = 6 drops to 41 of 70 with 8 jumps, and λ = 10 recovers 58 of 70 with none.

Worked example: one attention step

Character jα_prevf_jContent eα, content onlye with locationα, location-sensitive
b000.50.00550.50.0005
a (first)0.050.0550.49455.30.0555
n0.90.9250.50.00556.050.1176
a (second)0.050.550.49458.00.8264
Text "b a n a". The decoder has been reading n and the next frame is an a. Content energies β⟨q, k⟩ are 5 for both a's and 0.5 otherwise. Location feature f_j = α_prev(j) + 0.5 · α_prev(j − 1), weight λ = 6.

Content alone splits the weight 0.4945 / 0.4945 between the two a's, so the context vector is an average of the wrong letter and the right one. The location term adds 6×0.5=36\times0.5=3 to the second a, which is one step ahead of the current position, and only 6×0.05=0.36\times0.05=0.3 to the first a. After the softmax the right a gets 0.826. The n keeps 0.118 because the filter also rewards staying put. A learned model balances staying and advancing using the content scores.

python
1import torch, torch.nn as nn, torch.nn.functional as F
2
3class LocationSensitiveAttention(nn.Module):
4 def __init__(self, query_dim=1024, memory_dim=512, attn_dim=128, n_filters=32, kernel=31):
5 super().__init__()
6 self.W = nn.Linear(query_dim, attn_dim, bias=False)
7 self.V = nn.Linear(memory_dim, attn_dim, bias=True) # bias b inside the tanh
8 self.U = nn.Linear(n_filters, attn_dim, bias=False)
9 self.w = nn.Linear(attn_dim, 1, bias=False)
10 # channels: previous and cumulative weights (open-source practice; the paper names cumulative)
11 self.loc = nn.Conv1d(2, n_filters, kernel, padding=kernel // 2, bias=False)
12
13 def forward(self, s, h, prev, cum):
14 # s: (B, 1024) decoder state, h: (B, N, 512) encoder states, prev/cum: (B, N)
15 f = self.loc(torch.stack([prev, cum], dim=1)).transpose(1, 2) # (B, N, 32)
16 e = self.w(torch.tanh(self.W(s)[:, None] + self.V(h) + self.U(f))).squeeze(-1)
17 alpha = F.softmax(e, dim=-1) # (B, N)
18 context = torch.bmm(alpha[:, None], h).squeeze(1) # (B, 512)
19 return context, alpha, cum + alpha
20
21att = LocationSensitiveAttention()
22B, N = 2, 40
23h = torch.randn(B, N, 512)
24prev = torch.zeros(B, N); prev[:, 0] = 1.0 # start on character 0
25c, a, cum = att(torch.randn(B, 1024), h, prev, prev.clone())
26print(c.shape, a.shape) # torch.Size([2, 512]) torch.Size([2, 40])
Location-sensitive attention in PyTorch with Tacotron 2's sizes (not run here: torch was not available; the worked-example numbers were checked in numpy)

Use when

  • +Near-human naturalness: MOS 4.53 vs 4.58 for recordings in the paper's test
  • +Simple, modular blocks; the mel interface lets the vocoder be swapped (WaveRNN, WaveGlow, HiFi-GAN)
  • +Location-sensitive attention and a stop token make alignment and ending far more reliable than in Tacotron

Avoid when

  • −Autoregressive at two levels: about 80 decoder steps per second of audio, and the original WaveNet one sample at a time
  • −Soft attention can still skip or repeat on hard inputs
  • −Little direct control over speed or prosody

Check yourself

Content-based attention gives two identical letters exactly the same score. What lets location-sensitive attention pick the right one?

Check yourself

Tacotron 2's WaveNet receptive field is 6,139 samples at 24 kHz. Its mel frames have a 12.5 ms hop. About how many mel frames does one receptive field span?

Remember

  • ✓Encoder: 512-d character embedding → 3 conv layers (512 × width 5) → BiLSTM (256 + 256).
  • ✓Location-sensitive attention: e=w⊤tanh⁡(Ws+Vh+Uf+b)e=w^\top\tanh(Ws+Vh+Uf+b) with ff from 32 filters of length 31 over the cumulative attention weights.
  • ✓Decoder: dropout pre-net (kept on at inference) → 2 LSTMs of 1024 → one 80-band mel frame per step (r = 1), plus a stop token at p > 0.5.
  • ✓5-conv residual post-net; MSE before and after it. Mel: 50 ms window, 12.5 ms hop, 125 Hz–7.6 kHz.
  • ✓Modified WaveNet (30 layers, 3 cycles, 10-component mixture of logistics, 24 kHz): MOS 4.526 vs 4.582 for ground truth.
← back to Deep Learning roadmap