Luong attention

Luong, Pham and Manning (2015) simplified Bahdanau attention. They score the current decoder state hₜ against the encoder states (dot, general or concat), form h̃ₜ = tanh(W_c[cₜ; hₜ]), optionally attend only to a window around a predicted position (local attention), and feed h̃ₜ back as input.

13 min read1080 wordscore0 viewsupdated 2026-10-12

Definition

Luong attention (also multiplicative or global/local attention) is the family of attention layers from Luong, Pham and Manning, "Effective Approaches to Attention-based Neural Machine Translation" (EMNLP 2015). At target step tt the top-layer decoder state hth_t is scored against the top-layer encoder states hˉs\bar h_s. The softmax-weighted average gives the context ctc_t, and the prediction uses the attentional hidden state h~t=tanh⁡(Wc[ct;ht])\tilde h_t=\tanh(W_c[c_t;h_t]). Global attention looks at every source position. Local attention looks only at a window of 2D+12D+1 positions around an aligned position ptp_t.

Bahdanau attention had shown in 2014 that a decoder which searches the source beats a fixed-vector encoder–decoder. Its design, though, was one specific architecture: bidirectional encoder annotations, attention from the previous state si−1s_{i-1}, a single additive score, and a deep maxout output. Minh-Thang Luong, Hieu Pham and Christopher Manning took the stacked-LSTM sequence-to-sequence model of Sutskever et al. (see Sequence-to-sequence (Seq2Seq)) and asked which parts of attention matter. Their answer was a simpler computation path, three content-based score functions plus a location-based one, a cheaper local variant, and input feeding. On WMT'14 English→German an ensemble of their models reached 23.0 BLEU, above the best WMT'14 system (20.7).

Global attention

at(s)=exp⁡(score(ht,hˉs))∑s′exp⁡(score(ht,hˉs′)),ct=∑sat(s) hˉs,h~t=tanh⁡ ⁣(Wc[ct; ht]),p(yt∣y<t,x)=softmax(Wsh~t)a_t(s)=\frac{\exp\big(\mathrm{score}(h_t,\bar h_s)\big)}{\sum_{s'}\exp\big(\mathrm{score}(h_t,\bar h_{s'})\big)},\qquad c_t=\sum_s a_t(s)\,\bar h_s,\qquad \tilde h_t=\tanh\!\big(W_c[c_t;\,h_t]\big),\qquad p(y_t\mid y_{<t},x)=\mathrm{softmax}(W_s\tilde h_t)
hth_t
current top-layer decoder state, computed from yt−1y_{t-1} before attention
hˉs\bar h_s
top-layer encoder state at source position ss (unidirectional stacked LSTM in the paper)
ata_t
variable-length alignment vector over the SS source positions
Wc∈Rn×2nW_c\in\mathbb R^{n\times 2n}
combines context and state; h~t\tilde h_t is the "attentional vector" that both predicts yty_t and (with input feeding) is passed to step t+1t+1
WsW_s
output projection to the target vocabulary
score(ht,hˉs)={ht⊤hˉsdotht⊤Wahˉsgeneralva⊤tanh⁡ ⁣(Wa[ht;hˉs])concatat=softmax(Waht)  location\mathrm{score}(h_t,\bar h_s)=\begin{cases}h_t^\top\bar h_s & \textit{dot}\\[2pt] h_t^\top W_a\bar h_s & \textit{general}\\[2pt] v_a^\top\tanh\!\big(W_a[h_t;\bar h_s]\big) & \textit{concat}\end{cases}\qquad\qquad a_t=\mathrm{softmax}(W_ah_t)\ \ \textit{location}
dot\textit{dot}
no parameters; needs encoder and decoder states of the same size
general\textit{general}
a learned bilinear form Wa∈Rn×nW_a\in\mathbb R^{n\times n}; the "multiplicative" attention of later papers
concat\textit{concat}
the additive MLP of Bahdanau et al., written on the concatenation
location\textit{location}
weights computed from hth_t alone (no content), for a fixed maximum source length

Compare the order of operations. Luong: ht→at→ct→h~t→yth_t\to a_t\to c_t\to\tilde h_t\to y_t. Bahdanau: si−1→αi→ci→si→yis_{i-1}\to\alpha_i\to c_i\to s_i\to y_i. In Luong's version, attention sits on top of the recurrent stack, and the LSTM itself never sees the context unless it is fed back. Attention becomes a layer you can add to any stacked RNN decoder without touching the recurrence.

Local attention

Global attention scores every source word for every target word, which is expensive for long inputs such as paragraphs or documents. Inspired by the soft/hard trade-off of Xu et al. (2015), local attention first picks an aligned position ptp_t. It then attends softly only within [pt−D, pt+D][p_t-D,\,p_t+D], so at∈R2D+1a_t\in\mathbb R^{2D+1} has a fixed size. Windows that cross the sentence boundary are simply cut off. Two ways to choose ptp_t:

local-m:  pt=tlocal-p:  pt=S⋅sigmoid ⁣(vp⊤tanh⁡(Wpht)),at(s)=align(ht,hˉs) exp⁡ ⁣(−(s−pt)22σ2),  σ=D2\textit{local-m:}\ \ p_t=t\qquad\qquad \textit{local-p:}\ \ p_t=S\cdot\mathrm{sigmoid}\!\big(v_p^\top\tanh(W_ph_t)\big),\qquad a_t(s)=\mathrm{align}(h_t,\bar h_s)\,\exp\!\Big(-\frac{(s-p_t)^2}{2\sigma^2}\Big),\ \ \sigma=\frac D2
local-m\textit{local-m}
monotonic: assumes source and target are roughly aligned position by position
pt∈[0,S]p_t\in[0,S]
predicted real-valued position; WpW_p, vpv_p are learned, SS is the source length
align\mathrm{align}
the same softmax of scores as in global attention, taken over the window
exp⁡(⋅)\exp(\cdot)
a truncated Gaussian centred at ptp_t that favours positions near it; it is not renormalised, so the weights sum to less than 1
DD
window half-width, set empirically to 10 in the paper

The Gaussian is what makes local-p trainable. A hard window is piecewise constant in ptp_t, but the Gaussian factor is smooth: ∂at(s)/∂pt=at(s) (s−pt)/σ2\partial a_t(s)/\partial p_t=a_t(s)\,(s-p_t)/\sigma^2, and ∂pt/∂z=S sigmoid(z)(1−sigmoid(z))\partial p_t/\partial z=S\,\mathrm{sigmoid}(z)(1-\mathrm{sigmoid}(z)) for z=vp⊤tanh⁡(Wpht)z=v_p^\top\tanh(W_ph_t). Gradients push ptp_t toward window positions with useful content. The model is differentiable almost everywhere (the window edges move in integer jumps).

Input feeding

Global and local attention as written make each step's alignment decision independently. Phrase-based MT keeps a coverage set of source words already translated. Luong et al. approximate that by concatenating the previous attentional vector h~t−1\tilde h_{t-1} with the embedding of yt−1y_{t-1} as the input to the first LSTM layer, whose input size becomes 2n2n. The decoder now knows where it attended before, and the network becomes deep both horizontally and vertically. The cost is that the decoder has to be unrolled step by step even during training, because the input at tt depends on attention at t−1t-1.

Setup in the paper

Data
WMT'14 English↔German, 4.5M sentence pairs (116M English, 110M German words), 50K-word vocabularies, pairs longer than 50 words filtered
Model
4-layer stacked LSTMs, 1000 cells, 1000-dim embeddings, source reversed for the non-attentional baselines
Training
uniform init in [−0.1,0.1][-0.1,0.1], plain SGD with learning rate 1 for 10 epochs, halved every epoch after epoch 5; minibatch 128; gradient norm rescaled to 5; dropout 0.2 (12 epochs, halving after 8)
Local window
D=10D=10, σ=D/2=5\sigma=D/2=5
Speed
MATLAB on one Tesla K40: about 1K target words per second, 7–10 days per model
Extras
unknown-word replacement using the alignment (Luong et al. 2015, Jean et al. 2015), 8-model ensembles
aₜ(s) = align(hₜ, h̄ₛ) · exp(−(s − pₜ)²/2σ²), σ = D/2 · score = hₜᵀh̄ₛ (dot)wesawtheoldmanneartheriverandthebigblackdogranaway.nousavonsvulevieilhommeprèsdufleuveetlegroschiennoir┅ gold · ● pₜ · grey = outside the window (not scored) · click a row“le”: ■ aₜ(s) □ window softmax — Gaussiansource position s = 0 … 15mode local-ppₜ 2.0window [0, 4]gold s 2 (the)aₜ(gold) 0.909Σₛ aₜ(s) 0.923all 14 rows:mean aₜ(gold) 0.913argmax = gold 14/14scores 66 (global: 224)h̃ₜ = tanh(W_c[cₜ; hₜ]), 8 dimscₜhₜh̃ₜD = 2 · σ = 1.00 · predictor error 0.00 · window 2D+1 = 5 of 16 source words
attention

local-p here uses the gold position plus the error you set in place of the learned predictor pₜ = S·sigmoid(v_pᵀ tanh(W_p hₜ)).

fig.A 16-word English source and a 14-word French target. The states are noisy embeddings of the words, and the target 'chien noir' swaps 'black dog'. The source contains 'the' three times, so content alone cannot tell which copy a French article belongs to. Global dot attention (all 224 scores) puts 0.246 on the right 'the' for the first 'le' but 0.353 on the second copy. Two of 14 rows get the wrong argmax, and the mean weight on the gold word is 0.590. local-p with D = 2 scores only 66 pairs, gets 14/14 with mean gold weight 0.913, and the first 'le' gives 0.909 to the right copy. That row's weights sum to 0.923 because the Gaussian is not renormalised. local-m (pₜ = t) also gets 14/14 at D = 2. With D = 1 it misses 'noir', whose source word sits two positions behind t. With D = 4 the window holds two 'the' copies again (12/14). Add predictor error to local-p: at +1.5 the gold weight falls to 0.262, and at +3 every row misses (0/14). The right panel shows cₜ, hₜ and h̃ₜ = tanh(W_c[cₜ; hₜ]) for the selected row.

Worked example: global scores, h̃ₜ, and a local-p window

Decoder state ht=(1.2, 0.4)h_t=(1.2,\,0.4) and encoder states hˉ1=(1,0)\bar h_1=(1,0), hˉ2=(0.6,0.8)\bar h_2=(0.6,0.8), hˉ3=(−1,0.2)\bar h_3=(-1,0.2). For general, take Wa=(0110)W_a=\begin{pmatrix}0&1\\1&0\end{pmatrix} (it swaps the coordinates of hˉs\bar h_s).

sdot hₜᵀh̄ₛa (dot)general hₜᵀW_a h̄ₛa (general)
11.200.51270.400.2634
21.040.43691.200.5862
3−1.120.0504−0.160.1504
Global attention with two score functions (checked numerically in Node)

From weights to a prediction, then a local-p window

  1. 1Dot context: ct=0.5127 hˉ1+0.4369 hˉ2+0.0504 hˉ3=(0.7245, 0.3596)c_t=0.5127\,\bar h_1+0.4369\,\bar h_2+0.0504\,\bar h_3=(0.7245,\ 0.3596). The learned WaW_a of general would instead favour hˉ2\bar h_2.
  2. 2With Wc=(0.5−0.50.5001−0.50.5)W_c=\begin{pmatrix}0.5&-0.5&0.5&0\\0&1&-0.5&0.5\end{pmatrix}: Wc[ct;ht]=(0.7824, −0.0404)W_c[c_t;h_t]=(0.7824,\,-0.0404), so h~t=tanh⁡(⋅)=(0.6541, −0.0404)\tilde h_t=\tanh(\cdot)=(0.6541,\,-0.0404). This vector goes to softmax(Wsh~t)\mathrm{softmax}(W_s\tilde h_t) and, with input feeding, into step t+1t+1.
  3. 3Local-p in a 16-word sentence with z=vp⊤tanh⁡(Wpht)=0.2z=v_p^\top\tanh(W_ph_t)=0.2: pt=16⋅sigmoid(0.2)=8.797p_t=16\cdot\mathrm{sigmoid}(0.2)=8.797. With D=2D=2 (σ=1\sigma=1) the window is s=7,…,11s=7,\dots,11.
  4. 4If the window softmax is align=(0.1145, 0.1887, 0.5130, 0.1145, 0.0694)\mathrm{align}=(0.1145,\,0.1887,\,0.5130,\,0.1145,\,0.0694), the Gaussian factors are (0.1988, 0.7277, 0.9797, 0.4852, 0.0884)(0.1988,\,0.7277,\,0.9797,\,0.4852,\,0.0884). The final weights are (0.0228, 0.1373, 0.5025, 0.0555, 0.0061)(0.0228,\,0.1373,\,0.5025,\,0.0555,\,0.0061), summing to 0.72430.7243, not 1.

Results

systemperplexityBLEU
Winning WMT'14 system (phrase-based + large LM)20.7
Base (stacked LSTM)10.611.3
+ reverse source9.912.6
+ dropout8.114.0
+ global attention (location)7.316.8
+ input feeding6.418.1
local-p attention (general) + input feeding5.919.0
+ unknown-word replacement20.9
ensemble of 8 models + unk replacement23.0
WMT'14 English→German, newstest2014, tokenized BLEU (Luong et al. 2015, Table 1)

Attention plus input feeding took the dropout baseline from 14.0 to 19.0, the "+5.0 BLEU over non-attentional systems" of the abstract. On WMT'15 English→German the 8-model ensemble scored 25.9, then the state of the art. Their Table 4 compares architectures, before → after unknown-word replacement: global (location) 18.1 → 19.3, global (dot) 18.6 → 20.5, global (general) 17.3 → 19.1, local-m (general) 18.6 → 20.4, local-p (dot) 18.0 → 19.6, local-p (general) 19.0 → 20.9. Local-m with dot did not train well (perplexity above 7). In the authors' words, dot works well for global attention and general is better for local attention. Their concat implementation did not perform well (perplexities 6.7 to 7.1), possibly because they simplified WaW_a by fixing its hˉs\bar h_s block to the identity. Measured against gold alignments (Table 6, alignment error rate, lower is better), local attention aligned better than global: 0.34 for local-m and 0.36 for local-p, against 0.39 for global (location).

python
1import torch, torch.nn as nn, torch.nn.functional as F
2
3class LuongAttention(nn.Module):
4 def __init__(self, n, score="general", local_D=None):
5 super().__init__()
6 self.score, self.D = score, local_D
7 if score == "general":
8 self.Wa = nn.Linear(n, n, bias=False)
9 if score == "concat":
10 self.Wa = nn.Linear(2 * n, n, bias=False)
11 self.va = nn.Linear(n, 1, bias=False)
12 if local_D is not None: # local-p position predictor
13 self.Wp = nn.Linear(n, n, bias=False)
14 self.vp = nn.Linear(n, 1, bias=False)
15 self.Wc = nn.Linear(2 * n, n, bias=False)
16
17 def forward(self, h_t, Hs, mask):
18 # h_t (N, n) current decoder top state, Hs (N, S, n) encoder top states, mask (N, S) True = real
19 if self.score == "dot":
20 e = torch.bmm(Hs, h_t.unsqueeze(-1)).squeeze(-1)
21 elif self.score == "general":
22 e = torch.bmm(self.Wa(Hs), h_t.unsqueeze(-1)).squeeze(-1)
23 else:
24 e = self.va(torch.tanh(self.Wa(torch.cat([h_t.unsqueeze(1).expand_as(Hs), Hs], -1)))).squeeze(-1)
25 pos = torch.arange(Hs.size(1), device=Hs.device).float()
26 if self.D is not None:
27 S = mask.sum(-1, keepdim=True).float() # true source lengths
28 p_t = S * torch.sigmoid(self.vp(torch.tanh(self.Wp(h_t)))) # (N, 1), in [0, S]
29 window = (pos - p_t.round()).abs() <= self.D
30 mask = mask & window
31 a = e.masked_fill(~mask, float("-inf")).softmax(-1)
32 if self.D is not None:
33 sigma = self.D / 2
34 a = a * torch.exp(-(pos - p_t) ** 2 / (2 * sigma ** 2)) # not renormalised, as in the paper
35 c_t = torch.bmm(a.unsqueeze(1), Hs).squeeze(1)
36 h_tilde = torch.tanh(self.Wc(torch.cat([c_t, h_t], -1)))
37 return h_tilde, a
38
39class InputFeedingDecoder(nn.Module):
40 def __init__(self, v_tgt, n=1000, layers=4, score="general", local_D=10):
41 super().__init__()
42 self.emb = nn.Embedding(v_tgt, n, padding_idx=0)
43 self.rnn = nn.LSTM(2 * n, n, num_layers=layers, batch_first=True, dropout=0.2) # input = [y_{t-1}; h~_{t-1}]
44 self.attn = LuongAttention(n, score, local_D)
45 self.Ws = nn.Linear(n, v_tgt)
46
47 def forward(self, tgt_in, Hs, mask, state):
48 # state = (h_0, c_0) from the encoder, each (layers, N, n); teacher forcing over tgt_in
49 h_tilde = Hs.new_zeros(Hs.size(0), Hs.size(-1))
50 logits, aligns = [], []
51 for t in range(tgt_in.size(1)):
52 x = torch.cat([self.emb(tgt_in[:, t]), h_tilde], -1).unsqueeze(1)
53 out, state = self.rnn(x, state) # out[:, 0] is h_t (top layer)
54 h_tilde, a = self.attn(out[:, 0], Hs, mask) # h_t -> a_t -> c_t -> h~_t
55 logits.append(self.Ws(h_tilde))
56 aligns.append(a)
57 return torch.stack(logits, 1), torch.stack(aligns, 1), state
58
59N, S, T, n, L = 2, 16, 14, 64, 2
60enc = nn.LSTM(n, n, num_layers=L, batch_first=True)
61Hs, state = enc(torch.randn(N, S, n))
62mask = torch.ones(N, S, dtype=torch.bool); mask[1, 12:] = False
63dec = InputFeedingDecoder(v_tgt=500, n=n, layers=L, score="general", local_D=2)
64logits, aligns, _ = dec(torch.randint(1, 500, (N, T)), Hs, mask, state)
65print(logits.shape, aligns.shape) # torch.Size([2, 14, 500]) torch.Size([2, 14, 16])
66print((aligns > 0).sum(-1).max().item()) # at most 2D + 1 = 5 positions get weight
Luong attention (dot / general / concat, global or local-p) and an input-feeding decoder

Use when

  • +Simple, modular path ht→at→ct→h~th_t\to a_t\to c_t\to\tilde h_t on top of any stacked RNN
  • +dot and general are cheap batched matrix products
  • +Local attention: 2D+12D+1 scores per step instead of SS, and better alignment error rates in the paper
  • +Input feeding gives a learned form of coverage and improved BLEU by 1.3

Avoid when

  • −Local-p adds a position predictor and a window hyperparameter DD; a badly predicted ptp_t starves ctc_t
  • −Gaussian-weighted local weights do not form a distribution
  • −Input feeding forces step-by-step unrolling of the decoder even with teacher forcing
  • −concat underperformed in their implementation; dot requires equal state sizes

Check yourself

In Luong's global attention, what is the input to the output softmax at step t?

Check yourself

A local-p model predicts pₜ = 8.8 with D = 2. Which statement is true?

Remember

  • ✓Scores: dot ht⊤hˉsh_t^\top\bar h_s, general ht⊤Wahˉsh_t^\top W_a\bar h_s, concat va⊤tanh⁡(Wa[ht;hˉs])v_a^\top\tanh(W_a[h_t;\bar h_s]); location softmax(Waht)\mathrm{softmax}(W_ah_t).
  • ✓Uses the current decoder state: ht→at→ct→h~t=tanh⁡(Wc[ct;ht])→softmax(Wsh~t)h_t\to a_t\to c_t\to\tilde h_t=\tanh(W_c[c_t;h_t])\to\mathrm{softmax}(W_s\tilde h_t).
  • ✓Local attention: window [pt−D,pt+D][p_t-D,p_t+D]; local-m pt=tp_t=t; local-p pt=S sigmoid(vp⊤tanh⁡(Wpht))p_t=S\,\mathrm{sigmoid}(v_p^\top\tanh(W_ph_t)) with a Gaussian of σ=D/2\sigma=D/2 (unnormalised).
  • ✓Input feeding concatenates h~t−1\tilde h_{t-1} with the next input so alignment decisions can depend on past ones.
  • ✓WMT'14 En→De (Table 1): attention + input feeding 14.0 → 19.0 BLEU; 8-model ensemble 23.0 vs 20.7 for the best WMT'14 system.
← back to Deep Learning roadmap