Definition
Bahdanau attention (also additive or concat attention, the attention of RNNsearch) computes, for each target position , an energy between the previous decoder state and every encoder annotation . The softmax of the energies gives alignment weights , and the context is fed into the computation of the next decoder state and of the output word .
In 2014 neural machine translation meant an Encoder–decoder architecture: an RNN read the source sentence into one fixed-length vector, and a second RNN generated the translation from it (Cho et al., 2014; Sutskever, Vinyals and Le, 2014, see Sequence-to-sequence (Seq2Seq)). Cho et al. had shown that BLEU of such models falls steeply as sentences get longer. Dzmitry Bahdanau, Kyunghyun Cho and Yoshua Bengio's paper "Neural Machine Translation by Jointly Learning to Align and Translate" (arXiv September 2014, ICLR 2015) asked whether the vector was the problem. Their decoder searches the source for the relevant parts each time it emits a word. Alignment is not a separate latent variable or preprocessing step: it is a soft, differentiable weighting trained jointly with translation by backprop.
The model (RNNsearch)
- forward and backward encoder states at source word (a Bidirectional RNN); the annotation summarises the words on both sides of , with most weight on the words close to it
- decoder state just before emitting , size
- the alignment model: a one-hidden-layer MLP with tanh units applied to the pair
- energy: how well the input around position matches output position
- probability that is aligned to (translated from) ; row of the alignment matrix
- the "expected annotation" under the alignment distribution, a distinct context for every target word
- a gated recurrent unit (the GRU of Cho et al. 2014) whose candidate state and both gates all receive
- a "deep output": a single maxout hidden layer (500 units) over , the embedded and , then a softmax over the 30,000-word target shortlist
- initial decoder state from the backward encoder's first state, which has read the whole sentence
The order of operations matters. At step the model computes : attention is decided before the new state exists, and the context then enters the recurrence. Luong et al. (2015) later reversed this, attending from the current state (see Luong attention). The original also only uses the additive score. "Bahdanau attention" in libraries usually means just this score function, .
RNNsearch as published
- Task
- WMT'14 English→French, 348M words of parallel text after data selection, 30,000-word shortlists, [UNK] for the rest
- Sizes
- hidden units per encoder direction and in the decoder, embeddings , alignment hidden size , maxout layer
- Alignment parameters
- (: 1M, : 2M, : 1000)
- Init
- , and biases 0, so every alignment starts uniform; recurrent matrices random orthogonal
- Training
- minibatch SGD with Adadelta (, ), 80 sentences per batch, gradient L2 norm clipped at 1, about 5 days per model
- Decoding
- beam search for the most probable translation
- Cost
- once per sentence; per target word one and tanh-MLP evaluations, so for the whole sentence
Gradient of the alignment model
Let and , which the softmax gives as with . Backprop through the MLP is then:
- error at the tanh hidden units; zero when , so at initialisation only moves on the first step
- also flow through and (and through into ): attention gives every source word a direct gradient path from every target word
Worked example: one decoder step
Take and three annotations (each 2-dimensional: one forward and one backward unit): , , . The previous decoder state is , and , , .
| j | U_a h_j | W_a s + U_a h_j | tanh(·) | e_ij | α_ij |
|---|---|---|---|---|---|
| 1 | (0.80, 1.85) | (1.30, 0.85) | (0.8617, 0.6911) | 0.5162 | 0.4645 |
| 2 | (−0.60, 0.80) | (−0.10, −0.20) | (−0.0997, −0.1974) | −0.0010 | 0.2769 |
| 3 | (−1.10, −0.70) | (−0.60, −1.70) | (−0.5370, −0.9354) | −0.0693 | 0.2586 |
Step by step
- 1Query part, once per target step: . Key parts, once per sentence: in the second column.
- 2Add, squash, project: ; ; .
- 3Softmax: . Context , which the GRU then reads together with to produce .
- 4Precomputing is what makes the additive score affordable: only the -vector changes from step to step.
Results
| model | trained on sentences up to | BLEU, all | BLEU, no UNK |
|---|---|---|---|
| RNNencdec-30 | 30 words | 13.93 | 24.19 |
| RNNsearch-30 | 30 words | 21.50 | 31.44 |
| RNNencdec-50 | 50 words | 17.82 | 26.71 |
| RNNsearch-50 | 50 words | 26.75 | 34.16 |
| RNNsearch-50* (trained longer) | 50 words | 28.45 | 36.15 |
| Moses (phrase-based, + 418M-word LM) | n/a | 33.30 | 35.63 |
Two observations carried the paper. First, attention beat the fixed vector in every setting. RNNsearch-30 even beat RNNencdec-50 (21.50 vs 17.82), although it was trained on shorter sentences. Second, its BLEU-versus-length curve (their Figure 2) stays flat up to and beyond 50 words, where RNNencdec drops sharply. On sentences without unknown words, RNNsearch-50* (36.15) matched the phrase-based Moses system (35.63), which also used a separate 418M-word monolingual corpus. The remaining gap on all sentences comes largely from the 30k shortlist: words outside it become [UNK].
The alignment matrices (their Figure 3) are mostly diagonal, with characteristic exceptions. For "the European Economic Area" → "la zone économique européenne", the model aligned zone with Area, jumping over two words, and then looked back one word at a time. For "the man" → "l' homme", l' puts weight on both the and man, because the right French article depends on the following noun. A hard one-to-one alignment cannot express that.
1import torch, torch.nn as nn, torch.nn.functional as F23class AdditiveAttention(nn.Module):4 def __init__(self, dec_dim, enc_dim, attn_dim):5 super().__init__()6 self.W = nn.Linear(dec_dim, attn_dim, bias=False) # W_a7 self.U = nn.Linear(enc_dim, attn_dim, bias=False) # U_a8 self.v = nn.Linear(attn_dim, 1, bias=False) # v_a9 nn.init.zeros_(self.v.weight) # as in the paper: uniform attention at the start1011 def forward(self, s_prev, Uh, H, mask):12 # s_prev (N, n), Uh = U(H) (N, Tx, n'), H (N, Tx, 2n), mask (N, Tx) True = real token13 e = self.v(torch.tanh(self.W(s_prev).unsqueeze(1) + Uh)).squeeze(-1) # energies (N, Tx)14 alpha = e.masked_fill(~mask, float("-inf")).softmax(dim=-1)15 c = torch.bmm(alpha.unsqueeze(1), H).squeeze(1) # context (N, 2n)16 return c, alpha1718class RNNsearch(nn.Module):19 def __init__(self, v_src, v_tgt, m=620, n=1000, n_attn=1000):20 super().__init__()21 self.src_emb = nn.Embedding(v_src, m, padding_idx=0)22 self.tgt_emb = nn.Embedding(v_tgt, m, padding_idx=0)23 self.encoder = nn.GRU(m, n, batch_first=True, bidirectional=True)24 self.init_s = nn.Linear(n, n) # s_0 = tanh(W_s <-h_1)25 self.attn = AdditiveAttention(n, 2 * n, n_attn)26 self.cell = nn.GRUCell(m + 2 * n, n) # s_i = f(s_{i-1}, y_{i-1}, c_i)27 self.out = nn.Linear(n + m + 2 * n, v_tgt) # g(y_{i-1}, s_i, c_i); the paper adds a maxout layer2829 def forward(self, src, src_len, tgt_in):30 packed = nn.utils.rnn.pack_padded_sequence(self.src_emb(src), src_len.cpu(), batch_first=True, enforce_sorted=False)31 H, _ = self.encoder(packed) # packing: the backward GRU starts at each sentence's true end32 H, _ = nn.utils.rnn.pad_packed_sequence(H, batch_first=True, total_length=src.size(1))33 mask, n = src != 0, H.size(-1) // 234 s = torch.tanh(self.init_s(H[:, 0, n:])) # backward state at position 135 Uh = self.attn.U(H) # once per sentence36 logits, alphas = [], []37 for i in range(tgt_in.size(1)): # teacher forcing: tgt_in = <sos> y_1 ... y_{T-1}38 y_prev = self.tgt_emb(tgt_in[:, i])39 c, a = self.attn(s, Uh, H, mask) # attend with s_{i-1} ...40 s = self.cell(torch.cat([y_prev, c], dim=-1), s) # ... then update to s_i41 logits.append(self.out(torch.cat([s, y_prev, c], dim=-1)))42 alphas.append(a)43 return torch.stack(logits, 1), torch.stack(alphas, 1) # (N, Ty, V), (N, Ty, Tx)4445model = RNNsearch(v_src=100, v_tgt=120, m=32, n=64, n_attn=64)46src = torch.randint(1, 100, (2, 7)); src[1, 5:] = 0 # second sentence has 5 tokens + padding47src_len = torch.tensor([7, 5])48tgt_in, tgt_out = torch.randint(1, 120, (2, 6)), torch.randint(1, 120, (2, 6))49logits, alphas = model(src, src_len, tgt_in)50print(alphas.shape, alphas[1, :, 5:].sum().item()) # torch.Size([2, 6, 7]) 0.051print(alphas[0, 0]) # all 1/7: v_a = 0 at init5253loss = F.cross_entropy(logits.reshape(-1, 120), tgt_out.reshape(-1), ignore_index=0)54opt = torch.optim.Adadelta(model.parameters(), rho=0.95, eps=1e-6)55loss.backward()56torch.nn.utils.clip_grad_norm_(model.parameters(), max_norm=1.0) # the paper clips the gradient norm at 157opt.step()Use when
- +Removes the fixed-vector bottleneck: BLEU stays flat with sentence length
- +Learns soft word alignments as a by-product, with no alignment supervision
- +Additive score handles different query and key sizes ( vs ) and non-bilinear matching
- +Every source word gets a short gradient path from every target word
Avoid when
- − MLP evaluations per sentence; slower than dot-product scores on GPUs
- −Sequential decoder: attention at step depends on , so target positions cannot be processed in parallel
- −No memory of past alignments: prone to repeating or skipping source words
- −The 30k shortlist still limits quality on rare words ([UNK])
Check yourself
In Bahdanau attention, which decoder state is compared with the annotations to produce the alignment for target word yᵢ?
Check yourself
Why does the RNNsearch alignment start out uniform over the source words?
Remember
- ✓, , , with .
- ✓It uses the previous decoder state and feeds into the GRU update .
- ✓Precompute once per sentence; only changes per step.
- ✓WMT'14 En→Fr (Table 1): RNNsearch-50 26.75 vs RNNencdec-50 17.82 BLEU; RNNsearch-50* 36.15 on no-UNK sentences vs Moses 35.63.
- ✓The alignment matrix is learned without supervision; it is mostly diagonal, with jumps where the languages reorder.