# Bahdanau attention

> The first attention model for neural translation (Bahdanau, Cho and Bengio, 2014). Before emitting each word, the decoder scores every bidirectional-encoder annotation with a small MLP, vᵀtanh(W s₍ᵢ₋₁₎ + U hⱼ), and reads their softmax-weighted average. This removes the fixed-vector bottleneck.

Source: https://www.mlfoundry.in/learn/bahdanau-attention  
Section: Deep Learning › Recurrent Neural Networks and Sequence Models  
Level: core · 13 min read · updated 2026-10-12

**Bahdanau attention:** (also **additive** or **concat** attention, the attention of **RNNsearch**) computes, for each target position $i$, an energy $e_{ij}=v_a^\top\tanh(W_as_{i-1}+U_ah_j)$ between the **previous** decoder state $s_{i-1}$ and every encoder annotation $h_j$. The softmax of the energies gives alignment weights $\alpha_{ij}$, and the context $c_i=\sum_j\alpha_{ij}h_j$ is fed into the computation of the next decoder state $s_i$ and of the output word $y_i$.

In 2014 neural machine translation meant an **Encoder–decoder architecture**: an RNN read the source sentence into one fixed-length vector, and a second RNN generated the translation from it (Cho et al., 2014; Sutskever, Vinyals and Le, 2014, see **Sequence-to-sequence (Seq2Seq)**). Cho et al. had shown that BLEU of such models falls steeply as sentences get longer. Dzmitry Bahdanau, Kyunghyun Cho and Yoshua Bengio's paper "Neural Machine Translation by Jointly Learning to Align and Translate" (arXiv September 2014, ICLR 2015) asked whether the vector was the problem. Their decoder searches the source for the relevant parts each time it emits a word. Alignment is not a separate latent variable or preprocessing step: it is a soft, differentiable weighting trained jointly with translation by backprop.

## The model (RNNsearch)

$$
h_j=\big[\overrightarrow{h}_j;\ \overleftarrow{h}_j\big],\qquad e_{ij}=v_a^\top\tanh\!\big(W_a\,s_{i-1}+U_a\,h_j\big),\qquad \alpha_{ij}=\frac{\exp e_{ij}}{\sum_{k=1}^{T_x}\exp e_{ik}},\qquad c_i=\sum_{j=1}^{T_x}\alpha_{ij}\,h_j
$$

- $\overrightarrow{h}_j,\ \overleftarrow{h}_j$: forward and backward encoder states at source word $j$ (a **Bidirectional RNN**); the annotation $h_j\in\mathbb R^{2n}$ summarises the words on both sides of $x_j$, with most weight on the words close to it
- $s_{i-1}$: decoder state just **before** emitting $y_i$, size $n$
- $W_a\in\mathbb R^{n'\times n},\ U_a\in\mathbb R^{n'\times 2n},\ v_a\in\mathbb R^{n'}$: the alignment model: a one-hidden-layer MLP with $n'$ tanh units applied to the pair $(s_{i-1},h_j)$
- $e_{ij}$: energy: how well the input around position $j$ matches output position $i$
- $\alpha_{ij}$: probability that $y_i$ is aligned to (translated from) $x_j$; row $i$ of the alignment matrix
- $c_i$: the "expected annotation" under the alignment distribution, a distinct context for every target word

$$
s_i=f\big(s_{i-1},\,y_{i-1},\,c_i\big),\qquad p\big(y_i\mid y_{<i},x\big)=g\big(y_{i-1},\,s_i,\,c_i\big),\qquad s_0=\tanh\!\big(W_s\overleftarrow{h}_1\big)
$$

- $f$: a gated recurrent unit (the GRU of Cho et al. 2014) whose candidate state and both gates all receive $c_i$
- $g$: a "deep output": a single maxout hidden layer (500 units) over $s$, the embedded $y_{i-1}$ and $c_i$, then a softmax over the 30,000-word target shortlist
- $s_0$: initial decoder state from the backward encoder's first state, which has read the whole sentence

The order of operations matters. At step $i$ the model computes $s_{i-1}\to\alpha_i\to c_i\to s_i\to y_i$: attention is decided **before** the new state exists, and the context then enters the recurrence. Luong et al. (2015) later reversed this, attending from the current state $s_i$ (see **Luong attention**). The original also only uses the additive score. "Bahdanau attention" in libraries usually means just this score function, $v^\top\tanh(Wq+Uk)$.

**RNNsearch as published**

- **Task:** WMT'14 English→French, 348M words of parallel text after data selection, 30,000-word shortlists, [UNK] for the rest
- **Sizes:** $n=1000$ hidden units per encoder direction and in the decoder, embeddings $m=620$, alignment hidden size $n'=1000$, maxout layer $l=500$
- **Alignment parameters:** $n'(n+2n+1)=1000\cdot3001=3{,}001{,}000$ ($W_a$: 1M, $U_a$: 2M, $v_a$: 1000)
- **Init:** $W_a,U_a\sim\mathcal N(0,0.001^2)$, $v_a=0$ and biases 0, so every alignment starts uniform; recurrent matrices random orthogonal
- **Training:** minibatch SGD with Adadelta ($\epsilon=10^{-6}$, $\rho=0.95$), 80 sentences per batch, gradient L2 norm clipped at 1, about 5 days per model
- **Decoding:** beam search for the most probable translation
- **Cost:** $U_ah_j$ once per sentence; per target word one $W_as_{i-1}$ and $T_x$ tanh-MLP evaluations, so $O(T_xT_yn')$ for the whole sentence

## Gradient of the alignment model

Let $t_{ij}=\tanh(W_as_{i-1}+U_ah_j)$ and $\delta_{ij}=\partial L/\partial e_{ij}$, which the softmax gives as $\alpha_{ij}\big(g_{ij}-\sum_k\alpha_{ik}g_{ik}\big)$ with $g_{ij}=\partial L/\partial c_i\cdot h_j$. Backprop through the MLP is then:

$$
\frac{\partial L}{\partial v_a}=\sum_{i,j}\delta_{ij}\,t_{ij},\qquad \frac{\partial L}{\partial W_a}=\sum_{i}\Big(\sum_j\delta_{ij}\,v_a\odot(1-t_{ij}^2)\Big)s_{i-1}^\top,\qquad \frac{\partial L}{\partial U_a}=\sum_{j}\Big(\sum_i\delta_{ij}\,v_a\odot(1-t_{ij}^2)\Big)h_j^\top
$$

- $v_a\odot(1-t^2)$: error at the tanh hidden units; zero when $v_a=0$, so at initialisation only $v_a$ moves on the first step
- $\partial L/\partial s_{i-1},\ \partial L/\partial h_j$: also flow through $W_a^\top$ and $U_a^\top$ (and through $c_i$ into $h_j$): attention gives every source word a direct gradient path from every target word

> Figure: A toy RNNsearch trained live. The source is 6 distinct letters plus <eos>. A fixed bidirectional tanh RNN gives the annotations h_j, and a fixed decoder RNN over the previously emitted (teacher-forced) letters gives s_{i−1}. Only W_a, U_a, b_a and v_a are trained, by Adam on −log of the attention mass on the correct letter, with fresh random sentences each step. v_a starts at 0, so step 0 is uniform (α = 1/7, held-out loss ln 7 = 1.946). Press train: on 'reverse' the bidirectional encoder reaches 100% held-out alignment accuracy by step 100 and draws a clean anti-diagonal. With the forward half only it is stuck at 40.9% at step 200 and 69.0% at step 600. With the backward half only it reaches 84.9% and 99.5%. The reason is that the right source position j is the one whose right-hand neighbour was just emitted, and only ←h_j has read that neighbour. On 'copy' the roles swap: bidirectional 100% by step 200, forward only 88.0% and backward only 63.8% at step 600. Click a row to see its energies e_ij.

## Worked example: one decoder step

Take $n=n'=2$ and three annotations (each 2-dimensional: one forward and one backward unit): $h_1=(0.9,\,0.1)$, $h_2=(0.2,\,0.8)$, $h_3=(-0.5,\,0.6)$. The previous decoder state is $s_{i-1}=(0.5,\,-1)$, and $W_a=I$, $U_a=\begin{pmatrix}1&-1\\2&0.5\end{pmatrix}$, $v_a=(1,\,-0.5)$.

Additive attention by hand (checked numerically in Node)

| j | U_a h_j | W_a s + U_a h_j | tanh(·) | e_ij | α_ij |
| --- | --- | --- | --- | --- | --- |
| 1 | (0.80, 1.85) | (1.30, 0.85) | (0.8617, 0.6911) | 0.5162 | 0.4645 |
| 2 | (−0.60, 0.80) | (−0.10, −0.20) | (−0.0997, −0.1974) | −0.0010 | 0.2769 |
| 3 | (−1.10, −0.70) | (−0.60, −1.70) | (−0.5370, −0.9354) | −0.0693 | 0.2586 |

**Step by step**

1. Query part, once per target step: $W_as_{i-1}=(0.5,\,-1)$. Key parts, once per sentence: $U_ah_j$ in the second column.
2. Add, squash, project: $e_{i1}=0.8617-0.5\cdot0.6911=0.5162$; $e_{i2}=-0.0997+0.0987=-0.0010$; $e_{i3}=-0.5370+0.4677=-0.0693$.
3. Softmax: $\alpha_i=(0.4645,\ 0.2769,\ 0.2586)$. Context $c_i=\sum_j\alpha_{ij}h_j=(0.3441,\ 0.4232)$, which the GRU then reads together with $y_{i-1}$ to produce $s_i$.
4. Precomputing $U_ah_j$ is what makes the additive score affordable: only the $n'$-vector $W_as_{i-1}$ changes from step to step.

## Results

BLEU on the WMT'14 En→Fr test set (Bahdanau et al. 2015, Table 1). 'No UNK': only sentences without unknown words in source or reference.

| model | trained on sentences up to | BLEU, all | BLEU, no UNK |
| --- | --- | --- | --- |
| RNNencdec-30 | 30 words | 13.93 | 24.19 |
| RNNsearch-30 | 30 words | 21.50 | 31.44 |
| RNNencdec-50 | 50 words | 17.82 | 26.71 |
| RNNsearch-50 | 50 words | 26.75 | 34.16 |
| RNNsearch-50* (trained longer) | 50 words | 28.45 | 36.15 |
| Moses (phrase-based, + 418M-word LM) | n/a | 33.30 | 35.63 |

Two observations carried the paper. First, attention beat the fixed vector in every setting. RNNsearch-30 even beat RNNencdec-50 (21.50 vs 17.82), although it was trained on shorter sentences. Second, its BLEU-versus-length curve (their Figure 2) stays flat up to and beyond 50 words, where RNNencdec drops sharply. On sentences without unknown words, RNNsearch-50* (36.15) matched the phrase-based Moses system (35.63), which also used a separate 418M-word monolingual corpus. The remaining gap on all sentences comes largely from the 30k shortlist: words outside it become [UNK].

The alignment matrices (their Figure 3) are mostly diagonal, with characteristic exceptions. For "the European Economic Area" → "la zone économique européenne", the model aligned *zone* with *Area*, jumping over two words, and then looked back one word at a time. For "the man" → "l' homme", *l'* puts weight on both *the* and *man*, because the right French article depends on the following noun. A hard one-to-one alignment cannot express that.

```python
import torch, torch.nn as nn, torch.nn.functional as F

class AdditiveAttention(nn.Module):
    def __init__(self, dec_dim, enc_dim, attn_dim):
        super().__init__()
        self.W = nn.Linear(dec_dim, attn_dim, bias=False)   # W_a
        self.U = nn.Linear(enc_dim, attn_dim, bias=False)   # U_a
        self.v = nn.Linear(attn_dim, 1, bias=False)         # v_a
        nn.init.zeros_(self.v.weight)                       # as in the paper: uniform attention at the start

    def forward(self, s_prev, Uh, H, mask):
        # s_prev (N, n), Uh = U(H) (N, Tx, n'), H (N, Tx, 2n), mask (N, Tx) True = real token
        e = self.v(torch.tanh(self.W(s_prev).unsqueeze(1) + Uh)).squeeze(-1)   # energies (N, Tx)
        alpha = e.masked_fill(~mask, float("-inf")).softmax(dim=-1)
        c = torch.bmm(alpha.unsqueeze(1), H).squeeze(1)                        # context (N, 2n)
        return c, alpha

class RNNsearch(nn.Module):
    def __init__(self, v_src, v_tgt, m=620, n=1000, n_attn=1000):
        super().__init__()
        self.src_emb = nn.Embedding(v_src, m, padding_idx=0)
        self.tgt_emb = nn.Embedding(v_tgt, m, padding_idx=0)
        self.encoder = nn.GRU(m, n, batch_first=True, bidirectional=True)
        self.init_s = nn.Linear(n, n)                        # s_0 = tanh(W_s <-h_1)
        self.attn = AdditiveAttention(n, 2 * n, n_attn)
        self.cell = nn.GRUCell(m + 2 * n, n)                 # s_i = f(s_{i-1}, y_{i-1}, c_i)
        self.out = nn.Linear(n + m + 2 * n, v_tgt)           # g(y_{i-1}, s_i, c_i); the paper adds a maxout layer

    def forward(self, src, src_len, tgt_in):
        packed = nn.utils.rnn.pack_padded_sequence(self.src_emb(src), src_len.cpu(), batch_first=True, enforce_sorted=False)
        H, _ = self.encoder(packed)          # packing: the backward GRU starts at each sentence's true end
        H, _ = nn.utils.rnn.pad_packed_sequence(H, batch_first=True, total_length=src.size(1))
        mask, n = src != 0, H.size(-1) // 2
        s = torch.tanh(self.init_s(H[:, 0, n:]))            # backward state at position 1
        Uh = self.attn.U(H)                                  # once per sentence
        logits, alphas = [], []
        for i in range(tgt_in.size(1)):                      # teacher forcing: tgt_in = <sos> y_1 ... y_{T-1}
            y_prev = self.tgt_emb(tgt_in[:, i])
            c, a = self.attn(s, Uh, H, mask)                 # attend with s_{i-1} ...
            s = self.cell(torch.cat([y_prev, c], dim=-1), s) # ... then update to s_i
            logits.append(self.out(torch.cat([s, y_prev, c], dim=-1)))
            alphas.append(a)
        return torch.stack(logits, 1), torch.stack(alphas, 1)   # (N, Ty, V), (N, Ty, Tx)

model = RNNsearch(v_src=100, v_tgt=120, m=32, n=64, n_attn=64)
src = torch.randint(1, 100, (2, 7)); src[1, 5:] = 0          # second sentence has 5 tokens + padding
src_len = torch.tensor([7, 5])
tgt_in, tgt_out = torch.randint(1, 120, (2, 6)), torch.randint(1, 120, (2, 6))
logits, alphas = model(src, src_len, tgt_in)
print(alphas.shape, alphas[1, :, 5:].sum().item())          # torch.Size([2, 6, 7]) 0.0
print(alphas[0, 0])                                         # all 1/7: v_a = 0 at init

loss = F.cross_entropy(logits.reshape(-1, 120), tgt_out.reshape(-1), ignore_index=0)
opt = torch.optim.Adadelta(model.parameters(), rho=0.95, eps=1e-6)
loss.backward()
torch.nn.utils.clip_grad_norm_(model.parameters(), max_norm=1.0)   # the paper clips the gradient norm at 1
opt.step()
```

RNNsearch in PyTorch: bidirectional GRU encoder, additive attention on s_{i−1}, GRUCell decoder

> **Previous state, not current state:** Bahdanau attention queries with $s_{i-1}$ and feeds $c_i$ **into** the recurrence. Implementations that attend with the freshly computed $s_i$ and combine afterwards are Luong-style, even if they use the additive score. Both work, but the wiring changes what the GRU sees and how the context reaches the output. Be explicit about which one you mean when comparing results.

> **Masks and packing:** With padded batches, set the energies of padding positions to $-\infty$ before the softmax, or short sentences will attend to padding. Pack the source (`pack_padded_sequence`), so that the backward encoder direction starts at each sentence's real last word rather than at the padding. Otherwise $\overleftarrow{h}_j$, and the $s_0$ computed from it, are polluted by pad embeddings.

> **Why the bidirectional encoder matters:** The score sees one annotation at a time, so whatever context an output position needs to recognise its source word must already be inside $h_j$. The figure makes this concrete. To reverse a sequence, the decoder must find the word whose right-hand neighbour it has just written, and only the backward half of $h_j$ has read that neighbour. Real translation needs both sides, for agreement, articles and idioms, which is why RNNsearch concatenates the two directions (see **Bidirectional RNN**).

> **Coverage and later fixes:** Each step's attention is computed independently, so nothing stops the model from attending to the same source word twice (repetition) or never (omission). Tu et al. (2016) added a coverage vector, and Luong et al.'s input feeding passes past attentional states back into the decoder. For speech, Chorowski et al. (2015) made the additive score location-aware by adding convolutional features of the previous alignment $\alpha_{i-1}$ to its argument.

**Strengths**

- Removes the fixed-vector bottleneck: BLEU stays flat with sentence length
- Learns soft word alignments as a by-product, with no alignment supervision
- Additive score handles different query and key sizes ($n$ vs $2n$) and non-bilinear matching
- Every source word gets a short gradient path from every target word

**Limitations**

- $O(T_xT_y)$ MLP evaluations per sentence; slower than dot-product scores on GPUs
- Sequential decoder: attention at step $i$ depends on $s_{i-1}$, so target positions cannot be processed in parallel
- No memory of past alignments: prone to repeating or skipping source words
- The 30k shortlist still limits quality on rare words ([UNK])

**Question:** In Bahdanau attention, which decoder state is compared with the annotations to produce the alignment for target word yᵢ?

- A. sᵢ₋₁, the state before yᵢ is emitted
- B. sᵢ, the state after reading yᵢ₋₁ and cᵢ
- C. s₀ for every step
- D. The last encoder state

**Answer:** A. sᵢ₋₁, the state before yᵢ is emitted — $e_{ij}=a(s_{i-1},h_j)$. The context $c_i$ is then an input to $s_i=f(s_{i-1},y_{i-1},c_i)$. Luong attention instead scores with the current state $h_t$ and combines afterwards.

**Question:** Why does the RNNsearch alignment start out uniform over the source words?

- A. Because the source is reversed
- B. Because $v_a$ is initialised to 0, so every energy $e_{ij}$ is 0
- C. Because of dropout
- D. Because $W_a$ is the identity

**Answer:** B. Because $v_a$ is initialised to 0, so every energy $e_{ij}$ is 0 — With $v_a=0$, $e_{ij}=v_a^\top\tanh(\cdot)=0$ for all $j$, so $\alpha_{ij}=1/T_x$. The first gradient step moves only $v_a$, because the hidden-unit error $v_a\odot(1-t^2)$ is zero. In the figure, step 0 shows exactly 1/7 everywhere.

**Key points**

- $e_{ij}=v_a^\top\tanh(W_as_{i-1}+U_ah_j)$, $\alpha_{ij}=\mathrm{softmax}_j(e_{ij})$, $c_i=\sum_j\alpha_{ij}h_j$, with $h_j=[\overrightarrow h_j;\overleftarrow h_j]$.
- It uses the previous decoder state $s_{i-1}$ and feeds $c_i$ into the GRU update $s_i=f(s_{i-1},y_{i-1},c_i)$.
- Precompute $U_ah_j$ once per sentence; only $W_as_{i-1}$ changes per step.
- WMT'14 En→Fr (Table 1): RNNsearch-50 26.75 vs RNNencdec-50 17.82 BLEU; RNNsearch-50* 36.15 on no-UNK sentences vs Moses 35.63.
- The alignment matrix is learned without supervision; it is mostly diagonal, with jumps where the languages reorder.

---

From MLFoundry (https://www.mlfoundry.in). The web page adds interactive figures and diagrams.
