Definition
Tacotron is a sequence-to-sequence model with attention that maps a character string directly to a spectrogram, which a fixed algorithm (Griffin–Lim) turns into a waveform. It was introduced by Wang, Skerry-Ryan, Stanton, Wu, Weiss, Jaitly, Yang, Xiao, Chen, Bengio, Le, Agiomyrgiannakis, Clark and Saurous in "Tacotron: Towards End-to-End Speech Synthesis" (Interspeech 2017). Its successor is Tacotron 2.
The input is a few dozen characters. The output is thousands of spectrogram frames: one second at a 12.5 ms hop is 80 frames. Nothing says which characters produce which frames, and the same text can be spoken with many durations and intonations.
Earlier statistical parametric systems chained separately trained parts: a text front end that extracts linguistic features, a duration model, an acoustic model and a signal-processing vocoder. Each part needed expert design and its errors added up. Tacotron replaces all of them except the final phase reconstruction with one network trained on (text, audio) pairs. The pipeline stages it collapses are described in Text-to-speech.
Architecture
Characters are embedded as 256-d vectors and passed through a pre-net: two fully connected layers (256 then 128 units, ReLU) with dropout 0.5. The pre-net is a bottleneck that helps generalisation. The CBHG encoder then turns the sequence into one 256-d state per character.
CBHG stands for 1-D Convolution Bank, Highway network and bidirectional GRU. The bank applies sets of 128 filters, of widths , in parallel, so it sees unigram- to 16-gram-like patterns at once. The outputs are stacked, max-pooled along time with stride 1 (the length is kept), and projected back by two width-3 convolutions with a residual connection. Four Highway network layers and a bidirectional Gated recurrent unit (GRU) (128 units each way) then add sequence context.
Attention decoder and the reduction factor
The decoder is autoregressive. At step , its input is the previous output frame (an all-zero "<GO>" frame at the start), passed through its own pre-net. An attention GRU with 256 units produces a query , and Bahdanau attention (content-based, tanh) scores every encoder state against it. Two residual GRU layers (256 units) take the context vector and the query, and a linear layer outputs the next 80-band mel frames.
- attention-GRU state at decoder step (the query)
- encoder state of character (key and value)
- alignment: how much step reads character . Training should make it a monotonic diagonal
- context vector passed to the decoder GRUs
The scores depend only on content, not on position. Nothing stops step from reading a character far behind or ahead of where it was, which is the main weakness that Tacotron 2 fixes with location features.
The key trick is the reduction factor . Each decoder step predicts non-overlapping frames at once, and only the last one is fed back. Speech frames are highly correlated, and a character usually spans several frames, so little is lost. The paper used and reported that it sped up convergence and made the attention alignment form faster and more stably.
- number of spectrogram frames to generate
- frames emitted per decoder step (reduction factor)
- sequential decoder steps, and the number of rows in the alignment matrix
Post-net and Griffin–Lim
The decoder predicts mel frames left to right, so it never sees the future. A second CBHG, the post-processing net (), reads the whole predicted mel sequence in both directions and outputs a linear-frequency magnitude spectrogram: 1025 bins from a 2048-point FFT. Both outputs are trained with an L1 loss, weighted equally.
A magnitude spectrogram has no phase. Griffin–Lim (Griffin and Lim, 1984) estimates one by alternating between the time domain and the STFT domain, keeping the predicted magnitudes and updating only the phase (see Short-time Fourier transform (STFT)). Tacotron raised the predicted magnitudes to the power 1.2 first, which reduced artefacts, and ran 50 iterations.
- linear magnitude spectrogram predicted by the post-net
- phase of the current estimate; the magnitude is reset to every iteration
- initial phase (random or zero); the final waveform is of the last iterate
Setup and results (Wang et al., 2017)
- Data
- internal North American English corpus, about 24.6 hours from one professional female speaker
- Audio
- 24 kHz; Hann window of 50 ms, 12.5 ms hop, 2048-point FFT; pre-emphasis 0.97
- Targets
- 80-band mel spectrogram for the decoder (r = 2), linear spectrogram for the post-net; L1 loss on both
- Training
- batch 32, Adam; learning rate 0.001, decayed to 0.0005, 0.0003 and 0.0001 after 500k, 1M and 2M steps
- MOS
- 3.82 ± 0.085 on a 5-point scale, above a production parametric system (3.69) and below a production concatenative one (4.09)
Alignment and the reduction factor, live
Worked example: a 4-second utterance
| Reduction factor r | Frames per step | Decoder steps S = ⌈320 / r⌉ | Steps per second of audio |
|---|---|---|---|
| 1 | 1 | 320 | 80 |
| 2 (paper) | 2 | 160 | 40 |
| 3 | 3 | 107 | 26.75 |
| 5 | 5 | 64 | 16 |
At 12.5 ms and 24 kHz the hop is samples and the 50 ms window is 1200 samples, zero-padded to a 2048-point FFT. So frames. The decoder's mel target is numbers, while the post-net's linear target is , 12.8× more. Predicting the compact mel first and expanding it afterwards keeps the autoregressive part small.
1import numpy as np, librosa, scipy.signal, soundfile as sf23sr, n_fft, hop, win = 24000, 2048, 300, 1200 # 50 ms window, 12.5 ms hop4y, _ = librosa.load("speech.wav", sr=sr)5y = scipy.signal.lfilter([1, -0.97], [1], y) # pre-emphasis 0.9767S = np.abs(librosa.stft(y, n_fft=n_fft, hop_length=hop, win_length=win, window="hann"))8mel = librosa.feature.melspectrogram(S=S**2, sr=sr, n_mels=80)9print(S.shape, mel.shape) # (1025, T) post-net target, (80, T) decoder target1011# Pretend S came from the post-net: sharpen by 1.2, then recover a phase.12# momentum=0 gives classic Griffin-Lim; librosa's default 0.99 is the "fast" variant.13y_hat = librosa.griffinlim(S**1.2, n_iter=50, hop_length=hop, win_length=win,14 n_fft=n_fft, momentum=0.0)15y_hat = scipy.signal.lfilter([1], [1, -0.97], y_hat) # undo pre-emphasis16sf.write("tacotron_griffinlim.wav", y_hat / np.abs(y_hat).max(), sr)Use when
- +One network from characters to spectrogram; no hand-built linguistic features, duration model or aligner
- +Frame-level generation with r frames per step is far faster than sample-by-sample autoregressive synthesis
- +Trained from scratch on (text, audio) pairs with random initialisation
Avoid when
- −Griffin–Lim phase reconstruction leaves audible artefacts
- −Content-only attention can skip, repeat or stall, especially on long or unusual inputs
- −Still autoregressive over frames, so generation time grows with utterance length
Check yourself
A Tacotron model uses a 12.5 ms hop and r = 2. How many sequential decoder steps does it need for 3 seconds of speech?
Check yourself
Why does Tacotron's post-net predict a linear spectrogram rather than stopping at the mel frames?
Remember
- ✓Characters → embedding → pre-net → CBHG encoder (conv bank K = 16, highway, BiGRU) → one state per character.
- ✓Content-based tanh attention plus GRU decoders emit frames per step: steps; the paper used .
- ✓The CBHG post-net turns the mel sequence into a 1025-bin linear spectrogram; Griffin–Lim (power 1.2, 50 iterations) produces the 24 kHz waveform.
- ✓MOS 3.82 on internal US English data, above a parametric baseline (3.69) and below concatenative (4.09).
- ✓Its weak points, content-only attention and Griffin–Lim, are what Tacotron 2 replaced.