# Tacotron

> The first end-to-end neural TTS from characters: a CBHG encoder, content-based attention decoding r = 2 mel frames per step, a CBHG post-net to a linear spectrogram, and Griffin–Lim to audio. MOS 3.82.

Source: https://www.mlfoundry.in/learn/tacotron  
Section: Deep Learning › Speech and Audio Deep Learning  
Level: core · 8 min read · updated 2026-10-15

**Tacotron:** is a sequence-to-sequence model with attention that maps a character string directly to a spectrogram, which a fixed algorithm (Griffin–Lim) turns into a waveform. It was introduced by Wang, Skerry-Ryan, Stanton, Wu, Weiss, Jaitly, Yang, Xiao, Chen, Bengio, Le, Agiomyrgiannakis, Clark and Saurous in "Tacotron: Towards End-to-End Speech Synthesis" (Interspeech 2017). Its successor is **Tacotron 2**.

The input is a few dozen characters. The output is thousands of spectrogram frames: one second at a 12.5 ms hop is 80 frames. Nothing says which characters produce which frames, and the same text can be spoken with many durations and intonations.

Earlier statistical parametric systems chained separately trained parts: a text front end that extracts linguistic features, a duration model, an acoustic model and a signal-processing vocoder. Each part needed expert design and its errors added up. Tacotron replaces all of them except the final phase reconstruction with one network trained on (text, audio) pairs. The pipeline stages it collapses are described in **Text-to-speech**.

## Architecture

Characters are embedded as 256-d vectors and passed through a **pre-net**: two fully connected layers (256 then 128 units, ReLU) with dropout 0.5. The pre-net is a bottleneck that helps generalisation. The **CBHG** encoder then turns the sequence into one 256-d state $h_j$ per character.

> Figure: Tacotron as published. Characters pass through a pre-net and the CBHG encoder. Content-based attention, driven by a 256-unit attention GRU, feeds two residual GRUs that emit r = 2 mel frames per step. Only the last of those frames is fed back as the next input. A second CBHG maps the whole mel sequence to a 1025-bin linear spectrogram, and Griffin–Lim (magnitudes raised to 1.2, 50 iterations) recovers a 24 kHz waveform.

**CBHG** stands for 1-D **C**onvolution **B**ank, **H**ighway network and bidirectional **G**RU. The bank applies $K=16$ sets of 128 filters, of widths $k=1,\dots,16$, in parallel, so it sees unigram- to 16-gram-like patterns at once. The outputs are stacked, max-pooled along time with stride 1 (the length is kept), and projected back by two width-3 convolutions with a residual connection. Four **Highway network** layers and a bidirectional **Gated recurrent unit (GRU)** (128 units each way) then add sequence context.

## Attention decoder and the reduction factor

The decoder is autoregressive. At step $s$, its input is the previous output frame (an all-zero "<GO>" frame at the start), passed through its own pre-net. An attention GRU with 256 units produces a query $s_i$, and **Bahdanau attention** (content-based, tanh) scores every encoder state against it. Two residual GRU layers (256 units) take the context vector and the query, and a linear layer outputs the next 80-band mel frames.

$$
e_{i,j}=v^{\top}\tanh\!\big(W s_i+V h_j\big),\qquad \alpha_{i,j}=\frac{\exp e_{i,j}}{\sum_{k}\exp e_{i,k}},\qquad c_i=\sum_j \alpha_{i,j}\,h_j
$$

- $s_i$: attention-GRU state at decoder step $i$ (the query)
- $h_j$: encoder state of character $j$ (key and value)
- $\alpha_{i,j}$: alignment: how much step $i$ reads character $j$. Training should make it a monotonic diagonal
- $c_i$: context vector passed to the decoder GRUs

The scores depend only on content, not on position. Nothing stops step $i$ from reading a character far behind or ahead of where it was, which is the main weakness that **Tacotron 2** fixes with location features.

The key trick is the **reduction factor** $r$. Each decoder step predicts $r$ non-overlapping frames at once, and only the last one is fed back. Speech frames are highly correlated, and a character usually spans several frames, so little is lost. The paper used $r=2$ and reported that it sped up convergence and made the attention alignment form faster and more stably.

$$
S=\Big\lceil\frac{T}{r}\Big\rceil,\qquad T=\frac{\text{duration}}{\text{hop}}=\frac{4\ \text{s}}{12.5\ \text{ms}}=320\ \Rightarrow\ S_{r=2}=160
$$

- $T$: number of spectrogram frames to generate
- $r$: frames emitted per decoder step (reduction factor)
- $S$: sequential decoder steps, and the number of rows in the alignment matrix

## Post-net and Griffin–Lim

The decoder predicts mel frames left to right, so it never sees the future. A second CBHG, the **post-processing net** ($K=8$), reads the whole predicted mel sequence in both directions and outputs a **linear-frequency** magnitude spectrogram: 1025 bins from a 2048-point FFT. Both outputs are trained with an L1 loss, weighted equally.

A magnitude spectrogram has no phase. **Griffin–Lim** (Griffin and Lim, 1984) estimates one by alternating between the time domain and the STFT domain, keeping the predicted magnitudes and updating only the phase (see **Short-time Fourier transform (STFT)**). Tacotron raised the predicted magnitudes to the power 1.2 first, which reduced artefacts, and ran 50 iterations.

$$
X^{(k+1)}=\mathrm{STFT}\Big(\mathrm{ISTFT}\big(|\hat S|^{1.2}\, e^{\,i\angle X^{(k)}}\big)\Big),\qquad X^{(0)}=|\hat S|^{1.2}e^{\,i\phi_0}
$$

- $|\hat S|$: linear magnitude spectrogram predicted by the post-net
- $\angle X^{(k)}$: phase of the current estimate; the magnitude is reset to $|\hat S|^{1.2}$ every iteration
- $\phi_0$: initial phase (random or zero); the final waveform is $\mathrm{ISTFT}$ of the last iterate

**Setup and results (Wang et al., 2017)**

- **Data:** internal North American English corpus, about 24.6 hours from one professional female speaker
- **Audio:** 24 kHz; Hann window of 50 ms, 12.5 ms hop, 2048-point FFT; pre-emphasis 0.97
- **Targets:** 80-band mel spectrogram for the decoder (r = 2), linear spectrogram for the post-net; L1 loss on both
- **Training:** batch 32, Adam; learning rate 0.001, decayed to 0.0005, 0.0003 and 0.0001 after 500k, 1M and 2M steps
- **MOS:** 3.82 ± 0.085 on a 5-point scale, above a production parametric system (3.69) and below a production concatenative one (4.09)

## Alignment and the reduction factor, live

> Figure: A toy content-based attention on "the cat sat" (11 characters, 65 frames = 0.8125 s). The query of each step is the average of the frames it must emit. At the default r = 2 and β = 8 there are 33 decoder steps (3.00 per character), the argmax lands on a spoken character in 25 of 33 steps with 6 backward jumps, and the mean peak weight is 0.55. Most errors come from the repeated a and t, which content-only scores cannot tell apart. With r = 1 the decoder needs 65 steps (41 of 65 correct, 9 jumps). With r = 5 it needs 13 steps (1.18 per character, 11 of 13 correct, 2 jumps), so the diagonal is steeper and shorter. At β = 1 the attention is diffuse (mean peak 0.19), as in an untrained model.

## Worked example: a 4-second utterance

Decoder steps and target sizes for 4 s of 24 kHz audio (96,000 samples, hop 300 samples, window 1200 samples)

| Reduction factor r | Frames per step | Decoder steps S = ⌈320 / r⌉ | Steps per second of audio |
| --- | --- | --- | --- |
| 1 | 1 | 320 | 80 |
| 2 (paper) | 2 | 160 | 40 |
| 3 | 3 | 107 | 26.75 |
| 5 | 5 | 64 | 16 |

At 12.5 ms and 24 kHz the hop is $0.0125\times24{,}000=300$ samples and the 50 ms window is 1200 samples, zero-padded to a 2048-point FFT. So $T=96{,}000/300=320$ frames. The decoder's mel target is $320\times80=25{,}600$ numbers, while the post-net's linear target is $320\times1025=328{,}000$, 12.8× more. Predicting the compact mel first and expanding it afterwards keeps the autoregressive part small.

```python
import numpy as np, librosa, scipy.signal, soundfile as sf

sr, n_fft, hop, win = 24000, 2048, 300, 1200        # 50 ms window, 12.5 ms hop
y, _ = librosa.load("speech.wav", sr=sr)
y = scipy.signal.lfilter([1, -0.97], [1], y)         # pre-emphasis 0.97

S = np.abs(librosa.stft(y, n_fft=n_fft, hop_length=hop, win_length=win, window="hann"))
mel = librosa.feature.melspectrogram(S=S**2, sr=sr, n_mels=80)
print(S.shape, mel.shape)                            # (1025, T) post-net target, (80, T) decoder target

# Pretend S came from the post-net: sharpen by 1.2, then recover a phase.
# momentum=0 gives classic Griffin-Lim; librosa's default 0.99 is the "fast" variant.
y_hat = librosa.griffinlim(S**1.2, n_iter=50, hop_length=hop, win_length=win,
                           n_fft=n_fft, momentum=0.0)
y_hat = scipy.signal.lfilter([1], [1, -0.97], y_hat) # undo pre-emphasis
sf.write("tacotron_griffinlim.wav", y_hat / np.abs(y_hat).max(), sr)
```

Tacotron's targets and its Griffin–Lim step with librosa (not run here: librosa was not available; the shapes were checked separately)

> **Griffin–Lim caps the quality:** Griffin–Lim only finds a phase that is consistent with the magnitudes. It does not model how real speech phase behaves, so the output has a metallic, phasey quality even when the spectrogram is good. The authors noted these artefacts and left a better spectrogram-to-waveform inverter to future work. **Tacotron 2** replaced it with a neural vocoder (see **WaveNet**).

> **Why r helps the attention:** With $r=1$ the attention must hold still for about six steps on each character and then move by one. With larger $r$, each step covers more audio, so the alignment is a steeper diagonal with fewer rows. The recurrent decoder also has fewer steps to backpropagate through. The paper found that this made a usable alignment appear much earlier in training.

**Strengths**

- One network from characters to spectrogram; no hand-built linguistic features, duration model or aligner
- Frame-level generation with r frames per step is far faster than sample-by-sample autoregressive synthesis
- Trained from scratch on (text, audio) pairs with random initialisation

**Limitations**

- Griffin–Lim phase reconstruction leaves audible artefacts
- Content-only attention can skip, repeat or stall, especially on long or unusual inputs
- Still autoregressive over frames, so generation time grows with utterance length

**Question:** A Tacotron model uses a 12.5 ms hop and r = 2. How many sequential decoder steps does it need for 3 seconds of speech?

- A. 60
- B. 120
- C. 240
- D. 480

**Answer:** B. 120 — $3\text{ s}/12.5\text{ ms}=240$ frames, and $\lceil240/2\rceil=120$ steps. With $r=1$ it would be 240, with $r=5$ it would be 48.

**Question:** Why does Tacotron's post-net predict a linear spectrogram rather than stopping at the mel frames?

- A. Mel spectrograms cannot be computed at 24 kHz
- B. Griffin–Lim needs linear-frequency STFT magnitudes, and the post-net can use the whole sequence, including future frames, to produce them
- C. The linear spectrogram is smaller than the mel spectrogram
- D. The decoder cannot output more than 80 values per step

**Answer:** B. Griffin–Lim needs linear-frequency STFT magnitudes, and the post-net can use the whole sequence, including future frames, to produce them — Griffin–Lim inverts an STFT, so it needs all 1025 linear bins. The mel bank is not invertible exactly. The post-net is a non-causal CBHG, so it can correct each frame using frames on both sides, which the left-to-right decoder cannot do.

**Key points**

- Characters → embedding → pre-net → CBHG encoder (conv bank K = 16, highway, BiGRU) → one state per character.
- Content-based tanh attention plus GRU decoders emit $r$ frames per step: $S=\lceil T/r\rceil$ steps; the paper used $r=2$.
- The CBHG post-net turns the mel sequence into a 1025-bin linear spectrogram; Griffin–Lim (power 1.2, 50 iterations) produces the 24 kHz waveform.
- MOS 3.82 on internal US English data, above a parametric baseline (3.69) and below concatenative (4.09).
- Its weak points, content-only attention and Griffin–Lim, are what Tacotron 2 replaced.

---

From MLFoundry (https://www.mlfoundry.in). The web page adds interactive figures and diagrams.
