Definition
Luong attention (also multiplicative or global/local attention) is the family of attention layers from Luong, Pham and Manning, "Effective Approaches to Attention-based Neural Machine Translation" (EMNLP 2015). At target step the top-layer decoder state is scored against the top-layer encoder states . The softmax-weighted average gives the context , and the prediction uses the attentional hidden state . Global attention looks at every source position. Local attention looks only at a window of positions around an aligned position .
Bahdanau attention had shown in 2014 that a decoder which searches the source beats a fixed-vector encoder–decoder. Its design, though, was one specific architecture: bidirectional encoder annotations, attention from the previous state , a single additive score, and a deep maxout output. Minh-Thang Luong, Hieu Pham and Christopher Manning took the stacked-LSTM sequence-to-sequence model of Sutskever et al. (see Sequence-to-sequence (Seq2Seq)) and asked which parts of attention matter. Their answer was a simpler computation path, three content-based score functions plus a location-based one, a cheaper local variant, and input feeding. On WMT'14 English→German an ensemble of their models reached 23.0 BLEU, above the best WMT'14 system (20.7).
Global attention
- current top-layer decoder state, computed from before attention
- top-layer encoder state at source position (unidirectional stacked LSTM in the paper)
- variable-length alignment vector over the source positions
- combines context and state; is the "attentional vector" that both predicts and (with input feeding) is passed to step
- output projection to the target vocabulary
- no parameters; needs encoder and decoder states of the same size
- a learned bilinear form ; the "multiplicative" attention of later papers
- the additive MLP of Bahdanau et al., written on the concatenation
- weights computed from alone (no content), for a fixed maximum source length
Compare the order of operations. Luong: . Bahdanau: . In Luong's version, attention sits on top of the recurrent stack, and the LSTM itself never sees the context unless it is fed back. Attention becomes a layer you can add to any stacked RNN decoder without touching the recurrence.
Local attention
Global attention scores every source word for every target word, which is expensive for long inputs such as paragraphs or documents. Inspired by the soft/hard trade-off of Xu et al. (2015), local attention first picks an aligned position . It then attends softly only within , so has a fixed size. Windows that cross the sentence boundary are simply cut off. Two ways to choose :
- monotonic: assumes source and target are roughly aligned position by position
- predicted real-valued position; , are learned, is the source length
- the same softmax of scores as in global attention, taken over the window
- a truncated Gaussian centred at that favours positions near it; it is not renormalised, so the weights sum to less than 1
- window half-width, set empirically to 10 in the paper
The Gaussian is what makes local-p trainable. A hard window is piecewise constant in , but the Gaussian factor is smooth: , and for . Gradients push toward window positions with useful content. The model is differentiable almost everywhere (the window edges move in integer jumps).
Input feeding
Global and local attention as written make each step's alignment decision independently. Phrase-based MT keeps a coverage set of source words already translated. Luong et al. approximate that by concatenating the previous attentional vector with the embedding of as the input to the first LSTM layer, whose input size becomes . The decoder now knows where it attended before, and the network becomes deep both horizontally and vertically. The cost is that the decoder has to be unrolled step by step even during training, because the input at depends on attention at .
Setup in the paper
- Data
- WMT'14 English↔German, 4.5M sentence pairs (116M English, 110M German words), 50K-word vocabularies, pairs longer than 50 words filtered
- Model
- 4-layer stacked LSTMs, 1000 cells, 1000-dim embeddings, source reversed for the non-attentional baselines
- Training
- uniform init in , plain SGD with learning rate 1 for 10 epochs, halved every epoch after epoch 5; minibatch 128; gradient norm rescaled to 5; dropout 0.2 (12 epochs, halving after 8)
- Local window
- ,
- Speed
- MATLAB on one Tesla K40: about 1K target words per second, 7–10 days per model
- Extras
- unknown-word replacement using the alignment (Luong et al. 2015, Jean et al. 2015), 8-model ensembles
local-p here uses the gold position plus the error you set in place of the learned predictor pₜ = S·sigmoid(v_pᵀ tanh(W_p hₜ)).
Worked example: global scores, h̃ₜ, and a local-p window
Decoder state and encoder states , , . For general, take (it swaps the coordinates of ).
| s | dot hₜᵀh̄ₛ | a (dot) | general hₜᵀW_a h̄ₛ | a (general) |
|---|---|---|---|---|
| 1 | 1.20 | 0.5127 | 0.40 | 0.2634 |
| 2 | 1.04 | 0.4369 | 1.20 | 0.5862 |
| 3 | −1.12 | 0.0504 | −0.16 | 0.1504 |
From weights to a prediction, then a local-p window
- 1Dot context: . The learned of general would instead favour .
- 2With : , so . This vector goes to and, with input feeding, into step .
- 3Local-p in a 16-word sentence with : . With () the window is .
- 4If the window softmax is , the Gaussian factors are . The final weights are , summing to , not 1.
Results
| system | perplexity | BLEU |
|---|---|---|
| Winning WMT'14 system (phrase-based + large LM) | 20.7 | |
| Base (stacked LSTM) | 10.6 | 11.3 |
| + reverse source | 9.9 | 12.6 |
| + dropout | 8.1 | 14.0 |
| + global attention (location) | 7.3 | 16.8 |
| + input feeding | 6.4 | 18.1 |
| local-p attention (general) + input feeding | 5.9 | 19.0 |
| + unknown-word replacement | 20.9 | |
| ensemble of 8 models + unk replacement | 23.0 |
Attention plus input feeding took the dropout baseline from 14.0 to 19.0, the "+5.0 BLEU over non-attentional systems" of the abstract. On WMT'15 English→German the 8-model ensemble scored 25.9, then the state of the art. Their Table 4 compares architectures, before → after unknown-word replacement: global (location) 18.1 → 19.3, global (dot) 18.6 → 20.5, global (general) 17.3 → 19.1, local-m (general) 18.6 → 20.4, local-p (dot) 18.0 → 19.6, local-p (general) 19.0 → 20.9. Local-m with dot did not train well (perplexity above 7). In the authors' words, dot works well for global attention and general is better for local attention. Their concat implementation did not perform well (perplexities 6.7 to 7.1), possibly because they simplified by fixing its block to the identity. Measured against gold alignments (Table 6, alignment error rate, lower is better), local attention aligned better than global: 0.34 for local-m and 0.36 for local-p, against 0.39 for global (location).
1import torch, torch.nn as nn, torch.nn.functional as F23class LuongAttention(nn.Module):4 def __init__(self, n, score="general", local_D=None):5 super().__init__()6 self.score, self.D = score, local_D7 if score == "general":8 self.Wa = nn.Linear(n, n, bias=False)9 if score == "concat":10 self.Wa = nn.Linear(2 * n, n, bias=False)11 self.va = nn.Linear(n, 1, bias=False)12 if local_D is not None: # local-p position predictor13 self.Wp = nn.Linear(n, n, bias=False)14 self.vp = nn.Linear(n, 1, bias=False)15 self.Wc = nn.Linear(2 * n, n, bias=False)1617 def forward(self, h_t, Hs, mask):18 # h_t (N, n) current decoder top state, Hs (N, S, n) encoder top states, mask (N, S) True = real19 if self.score == "dot":20 e = torch.bmm(Hs, h_t.unsqueeze(-1)).squeeze(-1)21 elif self.score == "general":22 e = torch.bmm(self.Wa(Hs), h_t.unsqueeze(-1)).squeeze(-1)23 else:24 e = self.va(torch.tanh(self.Wa(torch.cat([h_t.unsqueeze(1).expand_as(Hs), Hs], -1)))).squeeze(-1)25 pos = torch.arange(Hs.size(1), device=Hs.device).float()26 if self.D is not None:27 S = mask.sum(-1, keepdim=True).float() # true source lengths28 p_t = S * torch.sigmoid(self.vp(torch.tanh(self.Wp(h_t)))) # (N, 1), in [0, S]29 window = (pos - p_t.round()).abs() <= self.D30 mask = mask & window31 a = e.masked_fill(~mask, float("-inf")).softmax(-1)32 if self.D is not None:33 sigma = self.D / 234 a = a * torch.exp(-(pos - p_t) ** 2 / (2 * sigma ** 2)) # not renormalised, as in the paper35 c_t = torch.bmm(a.unsqueeze(1), Hs).squeeze(1)36 h_tilde = torch.tanh(self.Wc(torch.cat([c_t, h_t], -1)))37 return h_tilde, a3839class InputFeedingDecoder(nn.Module):40 def __init__(self, v_tgt, n=1000, layers=4, score="general", local_D=10):41 super().__init__()42 self.emb = nn.Embedding(v_tgt, n, padding_idx=0)43 self.rnn = nn.LSTM(2 * n, n, num_layers=layers, batch_first=True, dropout=0.2) # input = [y_{t-1}; h~_{t-1}]44 self.attn = LuongAttention(n, score, local_D)45 self.Ws = nn.Linear(n, v_tgt)4647 def forward(self, tgt_in, Hs, mask, state):48 # state = (h_0, c_0) from the encoder, each (layers, N, n); teacher forcing over tgt_in49 h_tilde = Hs.new_zeros(Hs.size(0), Hs.size(-1))50 logits, aligns = [], []51 for t in range(tgt_in.size(1)):52 x = torch.cat([self.emb(tgt_in[:, t]), h_tilde], -1).unsqueeze(1)53 out, state = self.rnn(x, state) # out[:, 0] is h_t (top layer)54 h_tilde, a = self.attn(out[:, 0], Hs, mask) # h_t -> a_t -> c_t -> h~_t55 logits.append(self.Ws(h_tilde))56 aligns.append(a)57 return torch.stack(logits, 1), torch.stack(aligns, 1), state5859N, S, T, n, L = 2, 16, 14, 64, 260enc = nn.LSTM(n, n, num_layers=L, batch_first=True)61Hs, state = enc(torch.randn(N, S, n))62mask = torch.ones(N, S, dtype=torch.bool); mask[1, 12:] = False63dec = InputFeedingDecoder(v_tgt=500, n=n, layers=L, score="general", local_D=2)64logits, aligns, _ = dec(torch.randint(1, 500, (N, T)), Hs, mask, state)65print(logits.shape, aligns.shape) # torch.Size([2, 14, 500]) torch.Size([2, 14, 16])66print((aligns > 0).sum(-1).max().item()) # at most 2D + 1 = 5 positions get weightUse when
- +Simple, modular path on top of any stacked RNN
- +dot and general are cheap batched matrix products
- +Local attention: scores per step instead of , and better alignment error rates in the paper
- +Input feeding gives a learned form of coverage and improved BLEU by 1.3
Avoid when
- −Local-p adds a position predictor and a window hyperparameter ; a badly predicted starves
- −Gaussian-weighted local weights do not form a distribution
- −Input feeding forces step-by-step unrolling of the decoder even with teacher forcing
- −concat underperformed in their implementation; dot requires equal state sizes
Check yourself
In Luong's global attention, what is the input to the output softmax at step t?
Check yourself
A local-p model predicts pₜ = 8.8 with D = 2. Which statement is true?
Remember
- ✓Scores: dot , general , concat ; location .
- ✓Uses the current decoder state: .
- ✓Local attention: window ; local-m ; local-p with a Gaussian of (unnormalised).
- ✓Input feeding concatenates with the next input so alignment decisions can depend on past ones.
- ✓WMT'14 En→De (Table 1): attention + input feeding 14.0 → 19.0 BLEU; 8-model ensemble 23.0 vs 20.7 for the best WMT'14 system.