# Luong attention

> Luong, Pham and Manning (2015) simplified Bahdanau attention. They score the current decoder state hₜ against the encoder states (dot, general or concat), form h̃ₜ = tanh(W_c[cₜ; hₜ]), optionally attend only to a window around a predicted position (local attention), and feed h̃ₜ back as input.

Source: https://www.mlfoundry.in/learn/luong-attention  
Section: Deep Learning › Recurrent Neural Networks and Sequence Models  
Level: core · 13 min read · updated 2026-10-12

**Luong attention:** (also **multiplicative** or **global/local** attention) is the family of attention layers from Luong, Pham and Manning, "Effective Approaches to Attention-based Neural Machine Translation" (EMNLP 2015). At target step $t$ the top-layer decoder state $h_t$ is scored against the top-layer encoder states $\bar h_s$. The softmax-weighted average gives the context $c_t$, and the prediction uses the **attentional hidden state** $\tilde h_t=\tanh(W_c[c_t;h_t])$. **Global** attention looks at every source position. **Local** attention looks only at a window of $2D+1$ positions around an aligned position $p_t$.

**Bahdanau attention** had shown in 2014 that a decoder which searches the source beats a fixed-vector encoder–decoder. Its design, though, was one specific architecture: bidirectional encoder annotations, attention from the *previous* state $s_{i-1}$, a single additive score, and a deep maxout output. Minh-Thang Luong, Hieu Pham and Christopher Manning took the stacked-LSTM sequence-to-sequence model of Sutskever et al. (see **Sequence-to-sequence (Seq2Seq)**) and asked which parts of attention matter. Their answer was a simpler computation path, three content-based score functions plus a location-based one, a cheaper local variant, and **input feeding**. On WMT'14 English→German an ensemble of their models reached 23.0 BLEU, above the best WMT'14 system (20.7).

## Global attention

$$
a_t(s)=\frac{\exp\big(\mathrm{score}(h_t,\bar h_s)\big)}{\sum_{s'}\exp\big(\mathrm{score}(h_t,\bar h_{s'})\big)},\qquad c_t=\sum_s a_t(s)\,\bar h_s,\qquad \tilde h_t=\tanh\!\big(W_c[c_t;\,h_t]\big),\qquad p(y_t\mid y_{<t},x)=\mathrm{softmax}(W_s\tilde h_t)
$$

- $h_t$: **current** top-layer decoder state, computed from $y_{t-1}$ before attention
- $\bar h_s$: top-layer encoder state at source position $s$ (unidirectional stacked LSTM in the paper)
- $a_t$: variable-length alignment vector over the $S$ source positions
- $W_c\in\mathbb R^{n\times 2n}$: combines context and state; $\tilde h_t$ is the "attentional vector" that both predicts $y_t$ and (with input feeding) is passed to step $t+1$
- $W_s$: output projection to the target vocabulary

$$
\mathrm{score}(h_t,\bar h_s)=\begin{cases}h_t^\top\bar h_s & \textit{dot}\\[2pt] h_t^\top W_a\bar h_s & \textit{general}\\[2pt] v_a^\top\tanh\!\big(W_a[h_t;\bar h_s]\big) & \textit{concat}\end{cases}\qquad\qquad a_t=\mathrm{softmax}(W_ah_t)\ \ \textit{location}
$$

- $\textit{dot}$: no parameters; needs encoder and decoder states of the same size
- $\textit{general}$: a learned bilinear form $W_a\in\mathbb R^{n\times n}$; the "multiplicative" attention of later papers
- $\textit{concat}$: the additive MLP of Bahdanau et al., written on the concatenation
- $\textit{location}$: weights computed from $h_t$ alone (no content), for a fixed maximum source length

Compare the order of operations. Luong: $h_t\to a_t\to c_t\to\tilde h_t\to y_t$. Bahdanau: $s_{i-1}\to\alpha_i\to c_i\to s_i\to y_i$. In Luong's version, attention sits **on top of** the recurrent stack, and the LSTM itself never sees the context unless it is fed back. Attention becomes a layer you can add to any stacked RNN decoder without touching the recurrence.

## Local attention

Global attention scores every source word for every target word, which is expensive for long inputs such as paragraphs or documents. Inspired by the soft/hard trade-off of Xu et al. (2015), local attention first picks an **aligned position** $p_t$. It then attends softly only within $[p_t-D,\,p_t+D]$, so $a_t\in\mathbb R^{2D+1}$ has a fixed size. Windows that cross the sentence boundary are simply cut off. Two ways to choose $p_t$:

$$
\textit{local-m:}\ \ p_t=t\qquad\qquad \textit{local-p:}\ \ p_t=S\cdot\mathrm{sigmoid}\!\big(v_p^\top\tanh(W_ph_t)\big),\qquad a_t(s)=\mathrm{align}(h_t,\bar h_s)\,\exp\!\Big(-\frac{(s-p_t)^2}{2\sigma^2}\Big),\ \ \sigma=\frac D2
$$

- $\textit{local-m}$: monotonic: assumes source and target are roughly aligned position by position
- $p_t\in[0,S]$: predicted real-valued position; $W_p$, $v_p$ are learned, $S$ is the source length
- $\mathrm{align}$: the same softmax of scores as in global attention, taken over the window
- $\exp(\cdot)$: a truncated Gaussian centred at $p_t$ that favours positions near it; it is **not** renormalised, so the weights sum to less than 1
- $D$: window half-width, set empirically to 10 in the paper

The Gaussian is what makes local-p trainable. A hard window is piecewise constant in $p_t$, but the Gaussian factor is smooth: $\partial a_t(s)/\partial p_t=a_t(s)\,(s-p_t)/\sigma^2$, and $\partial p_t/\partial z=S\,\mathrm{sigmoid}(z)(1-\mathrm{sigmoid}(z))$ for $z=v_p^\top\tanh(W_ph_t)$. Gradients push $p_t$ toward window positions with useful content. The model is differentiable almost everywhere (the window edges move in integer jumps).

## Input feeding

Global and local attention as written make each step's alignment decision independently. Phrase-based MT keeps a **coverage set** of source words already translated. Luong et al. approximate that by concatenating the previous attentional vector $\tilde h_{t-1}$ with the embedding of $y_{t-1}$ as the input to the first LSTM layer, whose input size becomes $2n$. The decoder now knows where it attended before, and the network becomes deep both horizontally and vertically. The cost is that the decoder has to be unrolled step by step even during training, because the input at $t$ depends on attention at $t-1$.

**Setup in the paper**

- **Data:** WMT'14 English↔German, 4.5M sentence pairs (116M English, 110M German words), 50K-word vocabularies, pairs longer than 50 words filtered
- **Model:** 4-layer stacked LSTMs, 1000 cells, 1000-dim embeddings, source reversed for the non-attentional baselines
- **Training:** uniform init in $[-0.1,0.1]$, plain SGD with learning rate 1 for 10 epochs, halved every epoch after epoch 5; minibatch 128; gradient norm rescaled to 5; dropout 0.2 (12 epochs, halving after 8)
- **Local window:** $D=10$, $\sigma=D/2=5$
- **Speed:** MATLAB on one Tesla K40: about 1K target words per second, 7–10 days per model
- **Extras:** unknown-word replacement using the alignment (Luong et al. 2015, Jean et al. 2015), 8-model ensembles

> Figure: A 16-word English source and a 14-word French target. The states are noisy embeddings of the words, and the target 'chien noir' swaps 'black dog'. The source contains 'the' three times, so content alone cannot tell which copy a French article belongs to. Global dot attention (all 224 scores) puts 0.246 on the right 'the' for the first 'le' but 0.353 on the second copy. Two of 14 rows get the wrong argmax, and the mean weight on the gold word is 0.590. local-p with D = 2 scores only 66 pairs, gets 14/14 with mean gold weight 0.913, and the first 'le' gives 0.909 to the right copy. That row's weights sum to 0.923 because the Gaussian is not renormalised. local-m (pₜ = t) also gets 14/14 at D = 2. With D = 1 it misses 'noir', whose source word sits two positions behind t. With D = 4 the window holds two 'the' copies again (12/14). Add predictor error to local-p: at +1.5 the gold weight falls to 0.262, and at +3 every row misses (0/14). The right panel shows cₜ, hₜ and h̃ₜ = tanh(W_c[cₜ; hₜ]) for the selected row.

## Worked example: global scores, h̃ₜ, and a local-p window

Decoder state $h_t=(1.2,\,0.4)$ and encoder states $\bar h_1=(1,0)$, $\bar h_2=(0.6,0.8)$, $\bar h_3=(-1,0.2)$. For *general*, take $W_a=\begin{pmatrix}0&1\\1&0\end{pmatrix}$ (it swaps the coordinates of $\bar h_s$).

Global attention with two score functions (checked numerically in Node)

| s | dot hₜᵀh̄ₛ | a (dot) | general hₜᵀW_a h̄ₛ | a (general) |
| --- | --- | --- | --- | --- |
| 1 | 1.20 | 0.5127 | 0.40 | 0.2634 |
| 2 | 1.04 | 0.4369 | 1.20 | 0.5862 |
| 3 | −1.12 | 0.0504 | −0.16 | 0.1504 |

**From weights to a prediction, then a local-p window**

1. Dot context: $c_t=0.5127\,\bar h_1+0.4369\,\bar h_2+0.0504\,\bar h_3=(0.7245,\ 0.3596)$. The learned $W_a$ of *general* would instead favour $\bar h_2$.
2. With $W_c=\begin{pmatrix}0.5&-0.5&0.5&0\\0&1&-0.5&0.5\end{pmatrix}$: $W_c[c_t;h_t]=(0.7824,\,-0.0404)$, so $\tilde h_t=\tanh(\cdot)=(0.6541,\,-0.0404)$. This vector goes to $\mathrm{softmax}(W_s\tilde h_t)$ and, with input feeding, into step $t+1$.
3. Local-p in a 16-word sentence with $z=v_p^\top\tanh(W_ph_t)=0.2$: $p_t=16\cdot\mathrm{sigmoid}(0.2)=8.797$. With $D=2$ ($\sigma=1$) the window is $s=7,\dots,11$.
4. If the window softmax is $\mathrm{align}=(0.1145,\,0.1887,\,0.5130,\,0.1145,\,0.0694)$, the Gaussian factors are $(0.1988,\,0.7277,\,0.9797,\,0.4852,\,0.0884)$. The final weights are $(0.0228,\,0.1373,\,0.5025,\,0.0555,\,0.0061)$, summing to $0.7243$, not 1.

## Results

WMT'14 English→German, newstest2014, tokenized BLEU (Luong et al. 2015, Table 1)

| system | perplexity | BLEU |
| --- | --- | --- |
| Winning WMT'14 system (phrase-based + large LM) |  | 20.7 |
| Base (stacked LSTM) | 10.6 | 11.3 |
| + reverse source | 9.9 | 12.6 |
| + dropout | 8.1 | 14.0 |
| + global attention (location) | 7.3 | 16.8 |
| + input feeding | 6.4 | 18.1 |
| local-p attention (general) + input feeding | 5.9 | 19.0 |
| + unknown-word replacement |  | 20.9 |
| ensemble of 8 models + unk replacement |  | 23.0 |

Attention plus input feeding took the dropout baseline from 14.0 to 19.0, the "+5.0 BLEU over non-attentional systems" of the abstract. On WMT'15 English→German the 8-model ensemble scored 25.9, then the state of the art. Their Table 4 compares architectures, before → after unknown-word replacement: global (location) 18.1 → 19.3, global (dot) 18.6 → 20.5, global (general) 17.3 → 19.1, local-m (general) 18.6 → 20.4, local-p (dot) 18.0 → 19.6, local-p (general) 19.0 → 20.9. Local-m with dot did not train well (perplexity above 7). In the authors' words, *dot* works well for global attention and *general* is better for local attention. Their *concat* implementation did not perform well (perplexities 6.7 to 7.1), possibly because they simplified $W_a$ by fixing its $\bar h_s$ block to the identity. Measured against gold alignments (Table 6, alignment error rate, lower is better), local attention aligned better than global: 0.34 for local-m and 0.36 for local-p, against 0.39 for global (location).

```python
import torch, torch.nn as nn, torch.nn.functional as F

class LuongAttention(nn.Module):
    def __init__(self, n, score="general", local_D=None):
        super().__init__()
        self.score, self.D = score, local_D
        if score == "general":
            self.Wa = nn.Linear(n, n, bias=False)
        if score == "concat":
            self.Wa = nn.Linear(2 * n, n, bias=False)
            self.va = nn.Linear(n, 1, bias=False)
        if local_D is not None:                                  # local-p position predictor
            self.Wp = nn.Linear(n, n, bias=False)
            self.vp = nn.Linear(n, 1, bias=False)
        self.Wc = nn.Linear(2 * n, n, bias=False)

    def forward(self, h_t, Hs, mask):
        # h_t (N, n) current decoder top state, Hs (N, S, n) encoder top states, mask (N, S) True = real
        if self.score == "dot":
            e = torch.bmm(Hs, h_t.unsqueeze(-1)).squeeze(-1)
        elif self.score == "general":
            e = torch.bmm(self.Wa(Hs), h_t.unsqueeze(-1)).squeeze(-1)
        else:
            e = self.va(torch.tanh(self.Wa(torch.cat([h_t.unsqueeze(1).expand_as(Hs), Hs], -1)))).squeeze(-1)
        pos = torch.arange(Hs.size(1), device=Hs.device).float()
        if self.D is not None:
            S = mask.sum(-1, keepdim=True).float()                       # true source lengths
            p_t = S * torch.sigmoid(self.vp(torch.tanh(self.Wp(h_t))))  # (N, 1), in [0, S]
            window = (pos - p_t.round()).abs() <= self.D
            mask = mask & window
        a = e.masked_fill(~mask, float("-inf")).softmax(-1)
        if self.D is not None:
            sigma = self.D / 2
            a = a * torch.exp(-(pos - p_t) ** 2 / (2 * sigma ** 2))     # not renormalised, as in the paper
        c_t = torch.bmm(a.unsqueeze(1), Hs).squeeze(1)
        h_tilde = torch.tanh(self.Wc(torch.cat([c_t, h_t], -1)))
        return h_tilde, a

class InputFeedingDecoder(nn.Module):
    def __init__(self, v_tgt, n=1000, layers=4, score="general", local_D=10):
        super().__init__()
        self.emb = nn.Embedding(v_tgt, n, padding_idx=0)
        self.rnn = nn.LSTM(2 * n, n, num_layers=layers, batch_first=True, dropout=0.2)  # input = [y_{t-1}; h~_{t-1}]
        self.attn = LuongAttention(n, score, local_D)
        self.Ws = nn.Linear(n, v_tgt)

    def forward(self, tgt_in, Hs, mask, state):
        # state = (h_0, c_0) from the encoder, each (layers, N, n); teacher forcing over tgt_in
        h_tilde = Hs.new_zeros(Hs.size(0), Hs.size(-1))
        logits, aligns = [], []
        for t in range(tgt_in.size(1)):
            x = torch.cat([self.emb(tgt_in[:, t]), h_tilde], -1).unsqueeze(1)
            out, state = self.rnn(x, state)                      # out[:, 0] is h_t (top layer)
            h_tilde, a = self.attn(out[:, 0], Hs, mask)          # h_t -> a_t -> c_t -> h~_t
            logits.append(self.Ws(h_tilde))
            aligns.append(a)
        return torch.stack(logits, 1), torch.stack(aligns, 1), state

N, S, T, n, L = 2, 16, 14, 64, 2
enc = nn.LSTM(n, n, num_layers=L, batch_first=True)
Hs, state = enc(torch.randn(N, S, n))
mask = torch.ones(N, S, dtype=torch.bool); mask[1, 12:] = False
dec = InputFeedingDecoder(v_tgt=500, n=n, layers=L, score="general", local_D=2)
logits, aligns, _ = dec(torch.randint(1, 500, (N, T)), Hs, mask, state)
print(logits.shape, aligns.shape)                 # torch.Size([2, 14, 500]) torch.Size([2, 14, 16])
print((aligns > 0).sum(-1).max().item())          # at most 2D + 1 = 5 positions get weight
```

Luong attention (dot / general / concat, global or local-p) and an input-feeding decoder

> **Local-p weights do not sum to one:** The Gaussian multiplies an already normalised window softmax, so $\sum_s a_t(s)<1$, and $c_t$ shrinks whenever $p_t$ is off target (0.7243 in the worked example, 0.923 for the first 'le' in the figure). That is the paper's definition and part of the learning signal for $p_t$. Do not "fix" it by renormalising, or $p_t$ loses part of its gradient. If you want a proper distribution, put the Gaussian inside the softmax as a log-prior, $\mathrm{softmax}_s\big(\mathrm{score}-(s-p_t)^2/2\sigma^2\big)$, and say so.

> **local-m assumes monotonic, equal-length alignment:** $p_t=t$ is wrong as soon as the target is shorter or longer than the source, or reorders it. English→German verb movement is a standard example. The error grows along the sentence, and a word whose source falls outside $[t-D,t+D]$ cannot be attended at all ('noir' at $D=1$ in the figure). Local-p learns $p_t$ instead, and a moderate $D$ (10 in the paper) leaves room for error.

> **Why this design spread:** Placing attention after the recurrent stack, scoring with a matrix product, and combining with one $\tanh(W_c[c;h])$ layer turned attention into a plug-in layer over any encoder and decoder. *General* is $h_t^\top(W_a\bar h_s)$: compute $W_a\bar H$ once, and all scores for a step are one batched matrix–vector product. The Transformer's scaled dot-product attention (see **Self-Attention**) is the dot score with learned projections and a $1/\sqrt d$ factor. Input feeding is the RNN-era version of letting later steps see earlier attention decisions.

> **Which word does an alignment belong to?:** The paper's footnote 14: at step $t$ the model receives $y_{t-1}$, computes $h_t$ and $a_t$, then predicts $y_t$. For the content-based scores, the authors treat $a_t$ as the alignment of the **input** word $y_{t-1}$ when extracting alignments; for the location-based one, as that of the predicted word $y_t$. Keep this off-by-one in mind when you plot alignment matrices or replace unknown words with the aligned source word.

**Strengths**

- Simple, modular path $h_t\to a_t\to c_t\to\tilde h_t$ on top of any stacked RNN
- *dot* and *general* are cheap batched matrix products
- Local attention: $2D+1$ scores per step instead of $S$, and better alignment error rates in the paper
- Input feeding gives a learned form of coverage and improved BLEU by 1.3

**Limitations**

- Local-p adds a position predictor and a window hyperparameter $D$; a badly predicted $p_t$ starves $c_t$
- Gaussian-weighted local weights do not form a distribution
- Input feeding forces step-by-step unrolling of the decoder even with teacher forcing
- *concat* underperformed in their implementation; *dot* requires equal state sizes

**Question:** In Luong's global attention, what is the input to the output softmax at step t?

- A. The attentional state $\tilde h_t=\tanh(W_c[c_t;h_t])$
- B. The context $c_t$ alone
- C. The previous state $h_{t-1}$
- D. The encoder's last state

**Answer:** A. The attentional state $\tilde h_t=\tanh(W_c[c_t;h_t])$ — $p(y_t\mid\cdot)=\mathrm{softmax}(W_s\tilde h_t)$. The context and the current decoder state are combined by one tanh layer, and with input feeding the same $\tilde h_t$ is also fed into the next step's LSTM input.

**Question:** A local-p model predicts pₜ = 8.8 with D = 2. Which statement is true?

- A. All 16 source positions are scored, then a Gaussian is applied
- B. Only the 5 positions 7…11 are scored; their softmax weights are multiplied by exp(−(s − 8.8)²/2) and are not renormalised
- C. Only position 9 is used (hard attention)
- D. The weights are renormalised to sum to 1 after the Gaussian

**Answer:** B. Only the 5 positions 7…11 are scored; their softmax weights are multiplied by exp(−(s − 8.8)²/2) and are not renormalised — The window is $[p_t-D,p_t+D]$, here centred on 9: $2D+1=5$ positions. $\sigma=D/2=1$. The truncated Gaussian rescales the window softmax, so the weights sum to less than 1 (0.7243 in the worked example), and $p_t$ remains differentiable.

**Key points**

- Scores: dot $h_t^\top\bar h_s$, general $h_t^\top W_a\bar h_s$, concat $v_a^\top\tanh(W_a[h_t;\bar h_s])$; location $\mathrm{softmax}(W_ah_t)$.
- Uses the **current** decoder state: $h_t\to a_t\to c_t\to\tilde h_t=\tanh(W_c[c_t;h_t])\to\mathrm{softmax}(W_s\tilde h_t)$.
- Local attention: window $[p_t-D,p_t+D]$; local-m $p_t=t$; local-p $p_t=S\,\mathrm{sigmoid}(v_p^\top\tanh(W_ph_t))$ with a Gaussian of $\sigma=D/2$ (unnormalised).
- Input feeding concatenates $\tilde h_{t-1}$ with the next input so alignment decisions can depend on past ones.
- WMT'14 En→De (Table 1): attention + input feeding 14.0 → 19.0 BLEU; 8-model ensemble 23.0 vs 20.7 for the best WMT'14 system.

---

From MLFoundry (https://www.mlfoundry.in). The web page adds interactive figures and diagrams.
