Tacotron

The first end-to-end neural TTS from characters: a CBHG encoder, content-based attention decoding r = 2 mel frames per step, a CBHG post-net to a linear spectrogram, and Griffin–Lim to audio. MOS 3.82.

8 min read823 wordscore0 viewsupdated 2026-10-15

Definition

Tacotron is a sequence-to-sequence model with attention that maps a character string directly to a spectrogram, which a fixed algorithm (Griffin–Lim) turns into a waveform. It was introduced by Wang, Skerry-Ryan, Stanton, Wu, Weiss, Jaitly, Yang, Xiao, Chen, Bengio, Le, Agiomyrgiannakis, Clark and Saurous in "Tacotron: Towards End-to-End Speech Synthesis" (Interspeech 2017). Its successor is Tacotron 2.

The input is a few dozen characters. The output is thousands of spectrogram frames: one second at a 12.5 ms hop is 80 frames. Nothing says which characters produce which frames, and the same text can be spoken with many durations and intonations.

Earlier statistical parametric systems chained separately trained parts: a text front end that extracts linguistic features, a duration model, an acoustic model and a signal-processing vocoder. Each part needed expert design and its errors added up. Tacotron replaces all of them except the final phase reconstruction with one network trained on (text, audio) pairs. The pipeline stages it collapses are described in Text-to-speech.

Architecture

Characters are embedded as 256-d vectors and passed through a pre-net: two fully connected layers (256 then 128 units, ReLU) with dropout 0.5. The pre-net is a bottleneck that helps generalisation. The CBHG encoder then turns the sequence into one 256-d state hjh_j per character.

Tacotron (Wang et al., 2017) · encoder → attention decoder (r frames per step) → post-net → Griffin–Limcharacterst h e ␣ c a tone-hot→ 256-d embedpre-netFC-256, ReLUFC-128, ReLUdropout 0.5CBHG encoderConv1D bank k = 1…16 (128 ch each) → max-pool, stride 1→ 2 conv projections (width 3) + residual→ 4 highway layers → BiGRU 128 + 128encoder states h₁…h_N (256-d)decoder input<GO> frame at step 1,then the LAST of the rframes from step s − 1→ pre-net 256 → 128attentionattention GRU (256)content-based tanh:e = vᵀ tanh(W s + V h_j)context = Σ α_j h_jdecoder RNN2 residual GRUs (256)linear output:r × 80 mel valuesr = 2 → 2 frames/stepautoregressive: last predicted frame is the next step’s inputs1s2s3s480-band mel, 12.5 ms hoppost-net CBHGbank K = 8→ linear spectrum1025 bins (n_fft 2048)Griffin–Limmagnitudes ^ 1.250 iterationsSTFT ⇄ inverse STFTwaveform 24 kHzloss: L1 on the mel frames + L1 on the linear spectrogram · frames: 50 ms Hann window, 12.5 ms hop
fig.Tacotron as published. Characters pass through a pre-net and the CBHG encoder. Content-based attention, driven by a 256-unit attention GRU, feeds two residual GRUs that emit r = 2 mel frames per step. Only the last of those frames is fed back as the next input. A second CBHG maps the whole mel sequence to a 1025-bin linear spectrogram, and Griffin–Lim (magnitudes raised to 1.2, 50 iterations) recovers a 24 kHz waveform.

CBHG stands for 1-D Convolution Bank, Highway network and bidirectional GRU. The bank applies K=16K=16 sets of 128 filters, of widths k=1,…,16k=1,\dots,16, in parallel, so it sees unigram- to 16-gram-like patterns at once. The outputs are stacked, max-pooled along time with stride 1 (the length is kept), and projected back by two width-3 convolutions with a residual connection. Four Highway network layers and a bidirectional Gated recurrent unit (GRU) (128 units each way) then add sequence context.

Attention decoder and the reduction factor

The decoder is autoregressive. At step ss, its input is the previous output frame (an all-zero "<GO>" frame at the start), passed through its own pre-net. An attention GRU with 256 units produces a query sis_i, and Bahdanau attention (content-based, tanh) scores every encoder state against it. Two residual GRU layers (256 units) take the context vector and the query, and a linear layer outputs the next 80-band mel frames.

ei,j=v⊤tanh⁡ ⁣(Wsi+Vhj),αi,j=exp⁡ei,j∑kexp⁡ei,k,ci=∑jαi,j hje_{i,j}=v^{\top}\tanh\!\big(W s_i+V h_j\big),\qquad \alpha_{i,j}=\frac{\exp e_{i,j}}{\sum_{k}\exp e_{i,k}},\qquad c_i=\sum_j \alpha_{i,j}\,h_j
sis_i
attention-GRU state at decoder step ii (the query)
hjh_j
encoder state of character jj (key and value)
αi,j\alpha_{i,j}
alignment: how much step ii reads character jj. Training should make it a monotonic diagonal
cic_i
context vector passed to the decoder GRUs

The scores depend only on content, not on position. Nothing stops step ii from reading a character far behind or ahead of where it was, which is the main weakness that Tacotron 2 fixes with location features.

The key trick is the reduction factor rr. Each decoder step predicts rr non-overlapping frames at once, and only the last one is fed back. Speech frames are highly correlated, and a character usually spans several frames, so little is lost. The paper used r=2r=2 and reported that it sped up convergence and made the attention alignment form faster and more stably.

S=⌈Tr⌉,T=durationhop=4 s12.5 ms=320 ⇒ Sr=2=160S=\Big\lceil\frac{T}{r}\Big\rceil,\qquad T=\frac{\text{duration}}{\text{hop}}=\frac{4\ \text{s}}{12.5\ \text{ms}}=320\ \Rightarrow\ S_{r=2}=160
TT
number of spectrogram frames to generate
rr
frames emitted per decoder step (reduction factor)
SS
sequential decoder steps, and the number of rows in the alignment matrix

Post-net and Griffin–Lim

The decoder predicts mel frames left to right, so it never sees the future. A second CBHG, the post-processing net (K=8K=8), reads the whole predicted mel sequence in both directions and outputs a linear-frequency magnitude spectrogram: 1025 bins from a 2048-point FFT. Both outputs are trained with an L1 loss, weighted equally.

A magnitude spectrogram has no phase. Griffin–Lim (Griffin and Lim, 1984) estimates one by alternating between the time domain and the STFT domain, keeping the predicted magnitudes and updating only the phase (see Short-time Fourier transform (STFT)). Tacotron raised the predicted magnitudes to the power 1.2 first, which reduced artefacts, and ran 50 iterations.

X(k+1)=STFT(ISTFT(∣S^∣1.2 e i∠X(k))),X(0)=∣S^∣1.2e iϕ0X^{(k+1)}=\mathrm{STFT}\Big(\mathrm{ISTFT}\big(|\hat S|^{1.2}\, e^{\,i\angle X^{(k)}}\big)\Big),\qquad X^{(0)}=|\hat S|^{1.2}e^{\,i\phi_0}
∣S^∣|\hat S|
linear magnitude spectrogram predicted by the post-net
∠X(k)\angle X^{(k)}
phase of the current estimate; the magnitude is reset to ∣S^∣1.2|\hat S|^{1.2} every iteration
ϕ0\phi_0
initial phase (random or zero); the final waveform is ISTFT\mathrm{ISTFT} of the last iterate

Setup and results (Wang et al., 2017)

Data
internal North American English corpus, about 24.6 hours from one professional female speaker
Audio
24 kHz; Hann window of 50 ms, 12.5 ms hop, 2048-point FFT; pre-emphasis 0.97
Targets
80-band mel spectrogram for the decoder (r = 2), linear spectrogram for the post-net; L1 loss on both
Training
batch 32, Adam; learning rate 0.001, decayed to 0.0005, 0.0003 and 0.0001 after 500k, 1M and 2M steps
MOS
3.82 ± 0.085 on a 5-point scale, above a production parametric system (3.69) and below a production concatenative one (4.09)

Alignment and the reduction factor, live

target mel: 65 frames × 12.5 ms = 0.8125 s · r = 2 frames per decoder step → 33 stepsthe␣cat␣satmelticks: decoder steps · each step predicts 2 frames; only its last frame is fed back as the next inputattention α(step, character) = softmax(β · cos(query, key)) · outlined: characters spoken in that stepthe␣cat␣sat33 decoder steps for 11 characters (3.00 steps per character) · mean max α = 0.55argmax (dot) on a spoken character in 25/33 steps, red otherwise · backward jumps: 6content only: the two a’s and three t’s have near-identical keys, so a step can grab the wrong copy
β small: near-uniform attention, as at the start of training · β large: a sharp alignment
fig.A toy content-based attention on "the cat sat" (11 characters, 65 frames = 0.8125 s). The query of each step is the average of the frames it must emit. At the default r = 2 and β = 8 there are 33 decoder steps (3.00 per character), the argmax lands on a spoken character in 25 of 33 steps with 6 backward jumps, and the mean peak weight is 0.55. Most errors come from the repeated a and t, which content-only scores cannot tell apart. With r = 1 the decoder needs 65 steps (41 of 65 correct, 9 jumps). With r = 5 it needs 13 steps (1.18 per character, 11 of 13 correct, 2 jumps), so the diagonal is steeper and shorter. At β = 1 the attention is diffuse (mean peak 0.19), as in an untrained model.

Worked example: a 4-second utterance

Reduction factor rFrames per stepDecoder steps S = ⌈320 / r⌉Steps per second of audio
1132080
2 (paper)216040
3310726.75
556416
Decoder steps and target sizes for 4 s of 24 kHz audio (96,000 samples, hop 300 samples, window 1200 samples)

At 12.5 ms and 24 kHz the hop is 0.0125×24,000=3000.0125\times24{,}000=300 samples and the 50 ms window is 1200 samples, zero-padded to a 2048-point FFT. So T=96,000/300=320T=96{,}000/300=320 frames. The decoder's mel target is 320×80=25,600320\times80=25{,}600 numbers, while the post-net's linear target is 320×1025=328,000320\times1025=328{,}000, 12.8× more. Predicting the compact mel first and expanding it afterwards keeps the autoregressive part small.

python
1import numpy as np, librosa, scipy.signal, soundfile as sf
2
3sr, n_fft, hop, win = 24000, 2048, 300, 1200 # 50 ms window, 12.5 ms hop
4y, _ = librosa.load("speech.wav", sr=sr)
5y = scipy.signal.lfilter([1, -0.97], [1], y) # pre-emphasis 0.97
6
7S = np.abs(librosa.stft(y, n_fft=n_fft, hop_length=hop, win_length=win, window="hann"))
8mel = librosa.feature.melspectrogram(S=S**2, sr=sr, n_mels=80)
9print(S.shape, mel.shape) # (1025, T) post-net target, (80, T) decoder target
10
11# Pretend S came from the post-net: sharpen by 1.2, then recover a phase.
12# momentum=0 gives classic Griffin-Lim; librosa's default 0.99 is the "fast" variant.
13y_hat = librosa.griffinlim(S**1.2, n_iter=50, hop_length=hop, win_length=win,
14 n_fft=n_fft, momentum=0.0)
15y_hat = scipy.signal.lfilter([1], [1, -0.97], y_hat) # undo pre-emphasis
16sf.write("tacotron_griffinlim.wav", y_hat / np.abs(y_hat).max(), sr)
Tacotron's targets and its Griffin–Lim step with librosa (not run here: librosa was not available; the shapes were checked separately)

Use when

  • +One network from characters to spectrogram; no hand-built linguistic features, duration model or aligner
  • +Frame-level generation with r frames per step is far faster than sample-by-sample autoregressive synthesis
  • +Trained from scratch on (text, audio) pairs with random initialisation

Avoid when

  • −Griffin–Lim phase reconstruction leaves audible artefacts
  • −Content-only attention can skip, repeat or stall, especially on long or unusual inputs
  • −Still autoregressive over frames, so generation time grows with utterance length

Check yourself

A Tacotron model uses a 12.5 ms hop and r = 2. How many sequential decoder steps does it need for 3 seconds of speech?

Check yourself

Why does Tacotron's post-net predict a linear spectrogram rather than stopping at the mel frames?

Remember

  • ✓Characters → embedding → pre-net → CBHG encoder (conv bank K = 16, highway, BiGRU) → one state per character.
  • ✓Content-based tanh attention plus GRU decoders emit rr frames per step: S=⌈T/r⌉S=\lceil T/r\rceil steps; the paper used r=2r=2.
  • ✓The CBHG post-net turns the mel sequence into a 1025-bin linear spectrogram; Griffin–Lim (power 1.2, 50 iterations) produces the 24 kHz waveform.
  • ✓MOS 3.82 on internal US English data, above a parametric baseline (3.69) and below concatenative (4.09).
  • ✓Its weak points, content-only attention and Griffin–Lim, are what Tacotron 2 replaced.
← back to Deep Learning roadmap