# Tacotron 2

> Shen et al. (2018): a conv + BiLSTM encoder, location-sensitive attention and an LSTM decoder predict one 80-band mel frame per step plus a stop token; a modified WaveNet turns it into audio. MOS 4.53 vs 4.58 for recordings.

Source: https://www.mlfoundry.in/learn/tacotron-2  
Section: Deep Learning › Speech and Audio Deep Learning  
Level: core · 8 min read · updated 2026-10-15

**Tacotron 2:** is a two-part text-to-speech system. A sequence-to-sequence **spectrogram prediction network** maps characters to an 80-band mel spectrogram, and a modified **WaveNet** vocoder turns that spectrogram into a waveform. It was introduced by Shen, Pang, Weiss, Schuster, Jaitly, Yang, Chen, Zhang, Wang, Skerry-Ryan, Saurous, Agiomyrgiannakis and Wu in "Natural TTS Synthesis by Conditioning WaveNet on Mel Spectrogram Predictions" (ICASSP 2018).

**Tacotron** had shown that one network could map characters to spectrograms. However, its Griffin–Lim phase reconstruction sounded metallic, and its content-only attention could lose its place. Separately, WaveNet produced very natural audio, but in earlier systems it was conditioned on linguistic features, durations and F0 from a hand-built front end.

Tacotron 2 joins the two through a simple interface: the mel spectrogram. It is a compact (80 numbers per 12.5 ms frame), smooth target that is easy to predict with a squared loss. It still carries enough detail for a neural vocoder to fill in phase and fine structure. The general two-stage split is covered in **Text-to-speech**.

## Architecture

The **encoder** embeds each character as a 512-d vector. Three convolution layers (512 filters of width 5, **Batch normalization**, ReLU) model local context, like n-grams. A single **Bidirectional LSTM** with 256 units each way then gives one 512-d state $h_j$ per character. Compared with Tacotron's CBHG and GRUs, these are plain building blocks.

> Figure: Tacotron 2 as published. The encoder has 3 convolutions and a BiLSTM. Location-sensitive attention adds features computed by 32 filters of length 31 over the cumulative attention weights. The decoder has a pre-net (2 × 256 units, with dropout kept on at inference) and 2 LSTMs of 1024 units, and emits one 80-band mel frame and one stop probability per step. A 5-layer convolutional post-net adds a residual correction, and a modified WaveNet (30 dilated layers in 3 cycles, a 10-component mixture of logistics) produces 24 kHz, 16-bit audio.

## Location-sensitive attention

Content-based attention scores each character only by how well it matches the decoder state. If a word contains the same letter twice, or a phrase repeats, the scores cannot tell the copies apart. Tacotron 2 uses **location-sensitive attention** (Chorowski et al., 2015), which adds features computed from where the attention has already been.

$$
e_{i,j}=w^{\top}\tanh\!\big(W s_i+V h_j+U f_{i,j}+b\big),\qquad f_i=F*\textstyle\sum_{k<i}\alpha_k,\qquad \alpha_i=\mathrm{softmax}(e_i)
$$

- $s_i$: decoder LSTM state at step $i$ (the query)
- $h_j$: encoder state of character $j$
- $F$: 32 learned 1-D convolution filters of length 31, applied along the character axis to the cumulative attention weights
- $f_{i,j}$: 32-d location feature at character $j$. $W s$, $V h$ and $U f$ are projected to 128-d before the tanh
- $\alpha_i$: alignment over characters; context $c_i=\sum_j\alpha_{i,j}h_j$

The cumulative weights $\sum_{k<i}\alpha_k$ record how much each character has already been read. A filter can learn that the next character to read is just past the mass already consumed. This pushes the attention to move forward and makes repeating or skipping much less likely. Common open-source versions feed both the previous and the cumulative weights to the convolution.

## Decoder, post-net and stop token

The previous mel frame passes through a **pre-net** of two 256-unit ReLU layers with dropout 0.5. The dropout is also kept on at inference, which adds output variation. The pre-net output and the context vector feed two uni-directional **Long short-term memory (LSTM)** layers of 1024 units, regularised with zoneout 0.1. A linear projection of the LSTM output and the context gives the next frame. There is no reduction factor: one frame is produced per step. Training uses **Teacher forcing**, so the true previous frame is fed in.

$$
\hat y_{\text{post}}=\hat y+\mathrm{Postnet}(\hat y),\qquad \mathcal L=\lVert y-\hat y\rVert_2^2+\lVert y-\hat y_{\text{post}}\rVert_2^2,\qquad p^{\text{stop}}_i=\sigma\big(u^{\top}[s_i;c_i]+b_s\big)
$$

- $\hat y,\ \hat y_{\text{post}}$: mel frames before and after the post-net. The post-net has 5 convolutions with 512 filters of width 5, batch norm, and tanh on all but the last layer
- $\mathcal L$: summed mean squared error before and after the post-net (plus the stop-token loss)
- $p^{\text{stop}}_i$: probability that the utterance has ended. Generation stops at the first step with $p>0.5$, rather than after a fixed number of steps

The **vocoder** is a WaveNet with 30 dilated causal convolution layers in 3 cycles. The dilation of layer $k$ is $2^{k\bmod10}$, i.e. $1,2,\dots,512$ repeated three times. The mel frames are upsampled to the sample rate by 2 upsampling layers instead of 3. Instead of a 256-way softmax over μ-law values, it outputs a 10-component mixture of logistic distributions, which gives 16-bit samples at 24 kHz. It is trained on mel spectrograms that the prediction network produces with teacher forcing, so it learns to work with the predictor's slightly imperfect output.

Mean opinion scores on the paper's internal US English test set, 5-point scale (Shen et al., 2018)

| System | MOS |
| --- | --- |
| Parametric (production) | 3.492 ± 0.096 |
| Tacotron (Griffin–Lim) | 4.001 ± 0.087 |
| Concatenative (production) | 4.166 ± 0.091 |
| WaveNet (linguistic features) | 4.341 ± 0.051 |
| Tacotron 2 | 4.526 ± 0.066 |
| Ground-truth recordings | 4.582 ± 0.053 |

**Setup (Shen et al., 2018)**

- **Data:** internal US English corpus, 24.6 hours from one professional female speaker
- **Mel features:** 50 ms Hann window, 12.5 ms hop, 80 mel channels from 125 Hz to 7.6 kHz, log of the magnitude clipped below at 0.01
- **Training:** prediction network: teacher forcing, batch 64, Adam, learning rate 10⁻³ decaying to 10⁻⁵; WaveNet trained afterwards on its predictions
- **Vocoder:** 30 layers, 3 dilation cycles, receptive field 6,139 samples ≈ 255.8 ms at 24 kHz

## Content-only vs location-sensitive, live

> Figure: A toy decoder runs attention frame by frame over "banana bandana" (14 characters, 70 frames). Identical letters share one key, the hardest case. The location term is a fixed filter over the previous alignment that weights staying by 1 and moving one ahead by 0.5. At the default λ = 6 and σ = 0.3 the argmax is on the spoken character in 68 of 70 frames, with 0 backward jumps and no character skipped (mean peak 0.94). Press content only: 17 of 70 frames are correct, there are 9 backward jumps and 9 of the 14 characters are never attended. At frame 24, the first frame of the third letter, content-only attention spreads 0.16 over each of the six a's, while location-sensitive attention puts 0.78 on the right one. At σ = 0.6, λ = 6 drops to 41 of 70 with 8 jumps, and λ = 10 recovers 58 of 70 with none.

## Worked example: one attention step

Text "b a n a". The decoder has been reading n and the next frame is an a. Content energies β⟨q, k⟩ are 5 for both a's and 0.5 otherwise. Location feature f_j = α_prev(j) + 0.5 · α_prev(j − 1), weight λ = 6.

| Character j | α_prev | f_j | Content e | α, content only | e with location | α, location-sensitive |
| --- | --- | --- | --- | --- | --- | --- |
| b | 0 | 0 | 0.5 | 0.0055 | 0.5 | 0.0005 |
| a (first) | 0.05 | 0.05 | 5 | 0.4945 | 5.3 | 0.0555 |
| n | 0.9 | 0.925 | 0.5 | 0.0055 | 6.05 | 0.1176 |
| a (second) | 0.05 | 0.5 | 5 | 0.4945 | 8.0 | 0.8264 |

Content alone splits the weight 0.4945 / 0.4945 between the two a's, so the context vector is an average of the wrong letter and the right one. The location term adds $6\times0.5=3$ to the second a, which is one step ahead of the current position, and only $6\times0.05=0.3$ to the first a. After the softmax the right a gets 0.826. The n keeps 0.118 because the filter also rewards staying put. A learned model balances staying and advancing using the content scores.

```python
import torch, torch.nn as nn, torch.nn.functional as F

class LocationSensitiveAttention(nn.Module):
    def __init__(self, query_dim=1024, memory_dim=512, attn_dim=128, n_filters=32, kernel=31):
        super().__init__()
        self.W = nn.Linear(query_dim, attn_dim, bias=False)
        self.V = nn.Linear(memory_dim, attn_dim, bias=True)      # bias b inside the tanh
        self.U = nn.Linear(n_filters, attn_dim, bias=False)
        self.w = nn.Linear(attn_dim, 1, bias=False)
        # channels: previous and cumulative weights (open-source practice; the paper names cumulative)
        self.loc = nn.Conv1d(2, n_filters, kernel, padding=kernel // 2, bias=False)

    def forward(self, s, h, prev, cum):
        # s: (B, 1024) decoder state, h: (B, N, 512) encoder states, prev/cum: (B, N)
        f = self.loc(torch.stack([prev, cum], dim=1)).transpose(1, 2)        # (B, N, 32)
        e = self.w(torch.tanh(self.W(s)[:, None] + self.V(h) + self.U(f))).squeeze(-1)
        alpha = F.softmax(e, dim=-1)                                          # (B, N)
        context = torch.bmm(alpha[:, None], h).squeeze(1)                     # (B, 512)
        return context, alpha, cum + alpha

att = LocationSensitiveAttention()
B, N = 2, 40
h = torch.randn(B, N, 512)
prev = torch.zeros(B, N); prev[:, 0] = 1.0                                    # start on character 0
c, a, cum = att(torch.randn(B, 1024), h, prev, prev.clone())
print(c.shape, a.shape)       # torch.Size([2, 512]) torch.Size([2, 40])
```

Location-sensitive attention in PyTorch with Tacotron 2's sizes (not run here: torch was not available; the worked-example numbers were checked in numpy)

> **Attention can still fail:** Location features make the alignment much more reliable, but it is still learned soft attention. On very long sentences, unusual text or out-of-domain inputs, Tacotron 2 can still mumble, repeat or stop early. Parallel models such as **FastSpeech** remove this failure mode by predicting an explicit duration for each token.

> **Train the vocoder on predicted features:** If WaveNet were trained only on mel spectrograms computed from real audio, it would see slightly different inputs at synthesis time: the predictor's smoother, imperfect frames. Training it on the prediction network's teacher-forced outputs removes this mismatch. The same trick is used when fine-tuning later vocoders such as **HiFi-GAN** on an acoustic model's output.

**Strengths**

- Near-human naturalness: MOS 4.53 vs 4.58 for recordings in the paper's test
- Simple, modular blocks; the mel interface lets the vocoder be swapped (WaveRNN, WaveGlow, HiFi-GAN)
- Location-sensitive attention and a stop token make alignment and ending far more reliable than in Tacotron

**Limitations**

- Autoregressive at two levels: about 80 decoder steps per second of audio, and the original WaveNet one sample at a time
- Soft attention can still skip or repeat on hard inputs
- Little direct control over speed or prosody

**Question:** Content-based attention gives two identical letters exactly the same score. What lets location-sensitive attention pick the right one?

- A. A larger attention dimension (128 instead of 64)
- B. Convolutional features of the attention weights accumulated so far, which tell the scorer where the alignment currently is
- C. Bidirectional LSTMs in the decoder
- D. The stop token

**Answer:** B. Convolutional features of the attention weights accumulated so far, which tell the scorer where the alignment currently is — The term $U f_{i,j}$ is computed from $F*\sum_{k<i}\alpha_k$, so two characters with identical $h_j$ get different energies depending on their position relative to the attention already consumed. The usual learned result is to prefer the current or next character.

**Question:** Tacotron 2's WaveNet receptive field is 6,139 samples at 24 kHz. Its mel frames have a 12.5 ms hop. About how many mel frames does one receptive field span?

- A. About 2
- B. About 20
- C. About 200
- D. About 2,000

**Answer:** B. About 20 — One frame is $0.0125\times24{,}000=300$ samples, and $6139/300\approx20.5$ frames, or 255.8 ms. Each output sample sees about a quarter of a second of past audio, while the mel conditioning supplies the longer-range content.

**Key points**

- Encoder: 512-d character embedding → 3 conv layers (512 × width 5) → BiLSTM (256 + 256).
- Location-sensitive attention: $e=w^\top\tanh(Ws+Vh+Uf+b)$ with $f$ from 32 filters of length 31 over the cumulative attention weights.
- Decoder: dropout pre-net (kept on at inference) → 2 LSTMs of 1024 → one 80-band mel frame per step (r = 1), plus a stop token at p > 0.5.
- 5-conv residual post-net; MSE before and after it. Mel: 50 ms window, 12.5 ms hop, 125 Hz–7.6 kHz.
- Modified WaveNet (30 layers, 3 cycles, 10-component mixture of logistics, 24 kHz): MOS 4.526 vs 4.582 for ground truth.

---

From MLFoundry (https://www.mlfoundry.in). The web page adds interactive figures and diagrams.
