Bahdanau attention

The first attention model for neural translation (Bahdanau, Cho and Bengio, 2014). Before emitting each word, the decoder scores every bidirectional-encoder annotation with a small MLP, vᵀtanh(W s₍ᵢ₋₁₎ + U hⱼ), and reads their softmax-weighted average. This removes the fixed-vector bottleneck.

13 min read840 wordscore0 viewsupdated 2026-10-12

Definition

Bahdanau attention (also additive or concat attention, the attention of RNNsearch) computes, for each target position ii, an energy eij=va⊤tanh⁡(Wasi−1+Uahj)e_{ij}=v_a^\top\tanh(W_as_{i-1}+U_ah_j) between the previous decoder state si−1s_{i-1} and every encoder annotation hjh_j. The softmax of the energies gives alignment weights αij\alpha_{ij}, and the context ci=∑jαijhjc_i=\sum_j\alpha_{ij}h_j is fed into the computation of the next decoder state sis_i and of the output word yiy_i.

In 2014 neural machine translation meant an Encoder–decoder architecture: an RNN read the source sentence into one fixed-length vector, and a second RNN generated the translation from it (Cho et al., 2014; Sutskever, Vinyals and Le, 2014, see Sequence-to-sequence (Seq2Seq)). Cho et al. had shown that BLEU of such models falls steeply as sentences get longer. Dzmitry Bahdanau, Kyunghyun Cho and Yoshua Bengio's paper "Neural Machine Translation by Jointly Learning to Align and Translate" (arXiv September 2014, ICLR 2015) asked whether the vector was the problem. Their decoder searches the source for the relevant parts each time it emits a word. Alignment is not a separate latent variable or preprocessing step: it is a soft, differentiable weighting trained jointly with translation by backprop.

The model (RNNsearch)

hj=[h→j; h←j],eij=va⊤tanh⁡ ⁣(Wa si−1+Ua hj),αij=exp⁡eij∑k=1Txexp⁡eik,ci=∑j=1Txαij hjh_j=\big[\overrightarrow{h}_j;\ \overleftarrow{h}_j\big],\qquad e_{ij}=v_a^\top\tanh\!\big(W_a\,s_{i-1}+U_a\,h_j\big),\qquad \alpha_{ij}=\frac{\exp e_{ij}}{\sum_{k=1}^{T_x}\exp e_{ik}},\qquad c_i=\sum_{j=1}^{T_x}\alpha_{ij}\,h_j
h→j, h←j\overrightarrow{h}_j,\ \overleftarrow{h}_j
forward and backward encoder states at source word jj (a Bidirectional RNN); the annotation hj∈R2nh_j\in\mathbb R^{2n} summarises the words on both sides of xjx_j, with most weight on the words close to it
si−1s_{i-1}
decoder state just before emitting yiy_i, size nn
Wa∈Rn′×n, Ua∈Rn′×2n, va∈Rn′W_a\in\mathbb R^{n'\times n},\ U_a\in\mathbb R^{n'\times 2n},\ v_a\in\mathbb R^{n'}
the alignment model: a one-hidden-layer MLP with n′n' tanh units applied to the pair (si−1,hj)(s_{i-1},h_j)
eije_{ij}
energy: how well the input around position jj matches output position ii
αij\alpha_{ij}
probability that yiy_i is aligned to (translated from) xjx_j; row ii of the alignment matrix
cic_i
the "expected annotation" under the alignment distribution, a distinct context for every target word
si=f(si−1, yi−1, ci),p(yi∣y<i,x)=g(yi−1, si, ci),s0=tanh⁡ ⁣(Wsh←1)s_i=f\big(s_{i-1},\,y_{i-1},\,c_i\big),\qquad p\big(y_i\mid y_{<i},x\big)=g\big(y_{i-1},\,s_i,\,c_i\big),\qquad s_0=\tanh\!\big(W_s\overleftarrow{h}_1\big)
ff
a gated recurrent unit (the GRU of Cho et al. 2014) whose candidate state and both gates all receive cic_i
gg
a "deep output": a single maxout hidden layer (500 units) over ss, the embedded yi−1y_{i-1} and cic_i, then a softmax over the 30,000-word target shortlist
s0s_0
initial decoder state from the backward encoder's first state, which has read the whole sentence

The order of operations matters. At step ii the model computes si−1→αi→ci→si→yis_{i-1}\to\alpha_i\to c_i\to s_i\to y_i: attention is decided before the new state exists, and the context then enters the recurrence. Luong et al. (2015) later reversed this, attending from the current state sis_i (see Luong attention). The original also only uses the additive score. "Bahdanau attention" in libraries usually means just this score function, v⊤tanh⁡(Wq+Uk)v^\top\tanh(Wq+Uk).

RNNsearch as published

Task
WMT'14 English→French, 348M words of parallel text after data selection, 30,000-word shortlists, [UNK] for the rest
Sizes
n=1000n=1000 hidden units per encoder direction and in the decoder, embeddings m=620m=620, alignment hidden size n′=1000n'=1000, maxout layer l=500l=500
Alignment parameters
n′(n+2n+1)=1000⋅3001=3,001,000n'(n+2n+1)=1000\cdot3001=3{,}001{,}000 (WaW_a: 1M, UaU_a: 2M, vav_a: 1000)
Init
Wa,Ua∼N(0,0.0012)W_a,U_a\sim\mathcal N(0,0.001^2), va=0v_a=0 and biases 0, so every alignment starts uniform; recurrent matrices random orthogonal
Training
minibatch SGD with Adadelta (ϵ=10−6\epsilon=10^{-6}, ρ=0.95\rho=0.95), 80 sentences per batch, gradient L2 norm clipped at 1, about 5 days per model
Decoding
beam search for the most probable translation
Cost
UahjU_ah_j once per sentence; per target word one Wasi−1W_as_{i-1} and TxT_x tanh-MLP evaluations, so O(TxTyn′)O(T_xT_yn') for the whole sentence

Gradient of the alignment model

Let tij=tanh⁡(Wasi−1+Uahj)t_{ij}=\tanh(W_as_{i-1}+U_ah_j) and δij=∂L/∂eij\delta_{ij}=\partial L/\partial e_{ij}, which the softmax gives as αij(gij−∑kαikgik)\alpha_{ij}\big(g_{ij}-\sum_k\alpha_{ik}g_{ik}\big) with gij=∂L/∂ci⋅hjg_{ij}=\partial L/\partial c_i\cdot h_j. Backprop through the MLP is then:

∂L∂va=∑i,jδij tij,∂L∂Wa=∑i(∑jδij va⊙(1−tij2))si−1⊤,∂L∂Ua=∑j(∑iδij va⊙(1−tij2))hj⊤\frac{\partial L}{\partial v_a}=\sum_{i,j}\delta_{ij}\,t_{ij},\qquad \frac{\partial L}{\partial W_a}=\sum_{i}\Big(\sum_j\delta_{ij}\,v_a\odot(1-t_{ij}^2)\Big)s_{i-1}^\top,\qquad \frac{\partial L}{\partial U_a}=\sum_{j}\Big(\sum_i\delta_{ij}\,v_a\odot(1-t_{ij}^2)\Big)h_j^\top
va⊙(1−t2)v_a\odot(1-t^2)
error at the tanh hidden units; zero when va=0v_a=0, so at initialisation only vav_a moves on the first step
∂L/∂si−1, ∂L/∂hj\partial L/\partial s_{i-1},\ \partial L/\partial h_j
also flow through Wa⊤W_a^\top and Ua⊤U_a^\top (and through cic_i into hjh_j): attention gives every source word a direct gradient path from every target word
αᵢⱼ = softmaxⱼ(vₐᵀ tanh(Wₐ sᵢ₋₁ + Uₐ hⱼ)) · task: reverse · click a rowCHAEBF<eos>source xⱼ (annotation hⱼ)y1=F.14.14.14.14.14.14.14y2=B.14.14.14.14.14.14.14y3=E.14.14.14.14.14.14.14y4=A.14.14.14.14.14.14.14y5=H.14.14.14.14.14.14.14y6=C.14.14.14.14.14.14.14┅ gold alignment · darker = more weight · rows sum to 1i=2: sᵢ₋₁ reads <sos> Fxⱼenergy eᵢⱼweight αᵢⱼC0.000.143H0.000.143A0.000.143E0.000.143B0.000.143F0.000.143<eos>0.000.143context cᵢ = Σⱼ αᵢⱼ hⱼ · mass on gold 0.1430200400600100%0Adam steps · held-out alignment accuracy · held-out loss / ln 7step 0 · lr 0.03 · batch 16 · train loss — · held-out loss 1.946 · accuracy 16.7%
encoder
task
trainingonly Wₐ, Uₐ, bₐ, vₐ are trained; vₐ starts at 0, so step 0 is uniform (α = 1/7)
fig.A toy RNNsearch trained live. The source is 6 distinct letters plus <eos>. A fixed bidirectional tanh RNN gives the annotations h_j, and a fixed decoder RNN over the previously emitted (teacher-forced) letters gives s_{i−1}. Only W_a, U_a, b_a and v_a are trained, by Adam on −log of the attention mass on the correct letter, with fresh random sentences each step. v_a starts at 0, so step 0 is uniform (α = 1/7, held-out loss ln 7 = 1.946). Press train: on 'reverse' the bidirectional encoder reaches 100% held-out alignment accuracy by step 100 and draws a clean anti-diagonal. With the forward half only it is stuck at 40.9% at step 200 and 69.0% at step 600. With the backward half only it reaches 84.9% and 99.5%. The reason is that the right source position j is the one whose right-hand neighbour was just emitted, and only ←h_j has read that neighbour. On 'copy' the roles swap: bidirectional 100% by step 200, forward only 88.0% and backward only 63.8% at step 600. Click a row to see its energies e_ij.

Worked example: one decoder step

Take n=n′=2n=n'=2 and three annotations (each 2-dimensional: one forward and one backward unit): h1=(0.9, 0.1)h_1=(0.9,\,0.1), h2=(0.2, 0.8)h_2=(0.2,\,0.8), h3=(−0.5, 0.6)h_3=(-0.5,\,0.6). The previous decoder state is si−1=(0.5, −1)s_{i-1}=(0.5,\,-1), and Wa=IW_a=I, Ua=(1−120.5)U_a=\begin{pmatrix}1&-1\\2&0.5\end{pmatrix}, va=(1, −0.5)v_a=(1,\,-0.5).

jU_a h_jW_a s + U_a h_jtanh(·)e_ijα_ij
1(0.80, 1.85)(1.30, 0.85)(0.8617, 0.6911)0.51620.4645
2(−0.60, 0.80)(−0.10, −0.20)(−0.0997, −0.1974)−0.00100.2769
3(−1.10, −0.70)(−0.60, −1.70)(−0.5370, −0.9354)−0.06930.2586
Additive attention by hand (checked numerically in Node)

Step by step

  1. 1Query part, once per target step: Wasi−1=(0.5, −1)W_as_{i-1}=(0.5,\,-1). Key parts, once per sentence: UahjU_ah_j in the second column.
  2. 2Add, squash, project: ei1=0.8617−0.5⋅0.6911=0.5162e_{i1}=0.8617-0.5\cdot0.6911=0.5162; ei2=−0.0997+0.0987=−0.0010e_{i2}=-0.0997+0.0987=-0.0010; ei3=−0.5370+0.4677=−0.0693e_{i3}=-0.5370+0.4677=-0.0693.
  3. 3Softmax: αi=(0.4645, 0.2769, 0.2586)\alpha_i=(0.4645,\ 0.2769,\ 0.2586). Context ci=∑jαijhj=(0.3441, 0.4232)c_i=\sum_j\alpha_{ij}h_j=(0.3441,\ 0.4232), which the GRU then reads together with yi−1y_{i-1} to produce sis_i.
  4. 4Precomputing UahjU_ah_j is what makes the additive score affordable: only the n′n'-vector Wasi−1W_as_{i-1} changes from step to step.

Results

modeltrained on sentences up toBLEU, allBLEU, no UNK
RNNencdec-3030 words13.9324.19
RNNsearch-3030 words21.5031.44
RNNencdec-5050 words17.8226.71
RNNsearch-5050 words26.7534.16
RNNsearch-50* (trained longer)50 words28.4536.15
Moses (phrase-based, + 418M-word LM)n/a33.3035.63
BLEU on the WMT'14 En→Fr test set (Bahdanau et al. 2015, Table 1). 'No UNK': only sentences without unknown words in source or reference.

Two observations carried the paper. First, attention beat the fixed vector in every setting. RNNsearch-30 even beat RNNencdec-50 (21.50 vs 17.82), although it was trained on shorter sentences. Second, its BLEU-versus-length curve (their Figure 2) stays flat up to and beyond 50 words, where RNNencdec drops sharply. On sentences without unknown words, RNNsearch-50* (36.15) matched the phrase-based Moses system (35.63), which also used a separate 418M-word monolingual corpus. The remaining gap on all sentences comes largely from the 30k shortlist: words outside it become [UNK].

The alignment matrices (their Figure 3) are mostly diagonal, with characteristic exceptions. For "the European Economic Area" → "la zone économique européenne", the model aligned zone with Area, jumping over two words, and then looked back one word at a time. For "the man" → "l' homme", l' puts weight on both the and man, because the right French article depends on the following noun. A hard one-to-one alignment cannot express that.

python
1import torch, torch.nn as nn, torch.nn.functional as F
2
3class AdditiveAttention(nn.Module):
4 def __init__(self, dec_dim, enc_dim, attn_dim):
5 super().__init__()
6 self.W = nn.Linear(dec_dim, attn_dim, bias=False) # W_a
7 self.U = nn.Linear(enc_dim, attn_dim, bias=False) # U_a
8 self.v = nn.Linear(attn_dim, 1, bias=False) # v_a
9 nn.init.zeros_(self.v.weight) # as in the paper: uniform attention at the start
10
11 def forward(self, s_prev, Uh, H, mask):
12 # s_prev (N, n), Uh = U(H) (N, Tx, n'), H (N, Tx, 2n), mask (N, Tx) True = real token
13 e = self.v(torch.tanh(self.W(s_prev).unsqueeze(1) + Uh)).squeeze(-1) # energies (N, Tx)
14 alpha = e.masked_fill(~mask, float("-inf")).softmax(dim=-1)
15 c = torch.bmm(alpha.unsqueeze(1), H).squeeze(1) # context (N, 2n)
16 return c, alpha
17
18class RNNsearch(nn.Module):
19 def __init__(self, v_src, v_tgt, m=620, n=1000, n_attn=1000):
20 super().__init__()
21 self.src_emb = nn.Embedding(v_src, m, padding_idx=0)
22 self.tgt_emb = nn.Embedding(v_tgt, m, padding_idx=0)
23 self.encoder = nn.GRU(m, n, batch_first=True, bidirectional=True)
24 self.init_s = nn.Linear(n, n) # s_0 = tanh(W_s <-h_1)
25 self.attn = AdditiveAttention(n, 2 * n, n_attn)
26 self.cell = nn.GRUCell(m + 2 * n, n) # s_i = f(s_{i-1}, y_{i-1}, c_i)
27 self.out = nn.Linear(n + m + 2 * n, v_tgt) # g(y_{i-1}, s_i, c_i); the paper adds a maxout layer
28
29 def forward(self, src, src_len, tgt_in):
30 packed = nn.utils.rnn.pack_padded_sequence(self.src_emb(src), src_len.cpu(), batch_first=True, enforce_sorted=False)
31 H, _ = self.encoder(packed) # packing: the backward GRU starts at each sentence's true end
32 H, _ = nn.utils.rnn.pad_packed_sequence(H, batch_first=True, total_length=src.size(1))
33 mask, n = src != 0, H.size(-1) // 2
34 s = torch.tanh(self.init_s(H[:, 0, n:])) # backward state at position 1
35 Uh = self.attn.U(H) # once per sentence
36 logits, alphas = [], []
37 for i in range(tgt_in.size(1)): # teacher forcing: tgt_in = <sos> y_1 ... y_{T-1}
38 y_prev = self.tgt_emb(tgt_in[:, i])
39 c, a = self.attn(s, Uh, H, mask) # attend with s_{i-1} ...
40 s = self.cell(torch.cat([y_prev, c], dim=-1), s) # ... then update to s_i
41 logits.append(self.out(torch.cat([s, y_prev, c], dim=-1)))
42 alphas.append(a)
43 return torch.stack(logits, 1), torch.stack(alphas, 1) # (N, Ty, V), (N, Ty, Tx)
44
45model = RNNsearch(v_src=100, v_tgt=120, m=32, n=64, n_attn=64)
46src = torch.randint(1, 100, (2, 7)); src[1, 5:] = 0 # second sentence has 5 tokens + padding
47src_len = torch.tensor([7, 5])
48tgt_in, tgt_out = torch.randint(1, 120, (2, 6)), torch.randint(1, 120, (2, 6))
49logits, alphas = model(src, src_len, tgt_in)
50print(alphas.shape, alphas[1, :, 5:].sum().item()) # torch.Size([2, 6, 7]) 0.0
51print(alphas[0, 0]) # all 1/7: v_a = 0 at init
52
53loss = F.cross_entropy(logits.reshape(-1, 120), tgt_out.reshape(-1), ignore_index=0)
54opt = torch.optim.Adadelta(model.parameters(), rho=0.95, eps=1e-6)
55loss.backward()
56torch.nn.utils.clip_grad_norm_(model.parameters(), max_norm=1.0) # the paper clips the gradient norm at 1
57opt.step()
RNNsearch in PyTorch: bidirectional GRU encoder, additive attention on s_{i−1}, GRUCell decoder

Use when

  • +Removes the fixed-vector bottleneck: BLEU stays flat with sentence length
  • +Learns soft word alignments as a by-product, with no alignment supervision
  • +Additive score handles different query and key sizes (nn vs 2n2n) and non-bilinear matching
  • +Every source word gets a short gradient path from every target word

Avoid when

  • −O(TxTy)O(T_xT_y) MLP evaluations per sentence; slower than dot-product scores on GPUs
  • −Sequential decoder: attention at step ii depends on si−1s_{i-1}, so target positions cannot be processed in parallel
  • −No memory of past alignments: prone to repeating or skipping source words
  • −The 30k shortlist still limits quality on rare words ([UNK])

Check yourself

In Bahdanau attention, which decoder state is compared with the annotations to produce the alignment for target word yᵢ?

Check yourself

Why does the RNNsearch alignment start out uniform over the source words?

Remember

  • ✓eij=va⊤tanh⁡(Wasi−1+Uahj)e_{ij}=v_a^\top\tanh(W_as_{i-1}+U_ah_j), αij=softmaxj(eij)\alpha_{ij}=\mathrm{softmax}_j(e_{ij}), ci=∑jαijhjc_i=\sum_j\alpha_{ij}h_j, with hj=[h→j;h←j]h_j=[\overrightarrow h_j;\overleftarrow h_j].
  • ✓It uses the previous decoder state si−1s_{i-1} and feeds cic_i into the GRU update si=f(si−1,yi−1,ci)s_i=f(s_{i-1},y_{i-1},c_i).
  • ✓Precompute UahjU_ah_j once per sentence; only Wasi−1W_as_{i-1} changes per step.
  • ✓WMT'14 En→Fr (Table 1): RNNsearch-50 26.75 vs RNNencdec-50 17.82 BLEU; RNNsearch-50* 36.15 on no-UNK sentences vs Moses 35.63.
  • ✓The alignment matrix is learned without supervision; it is mostly diagonal, with jumps where the languages reorder.
← back to Deep Learning roadmap