Definition
Transformer-XL is a decoder-only language model that processes a long text as consecutive segments of tokens and adds segment-level recurrence. After a segment is processed, every layer's hidden states are stored as memory. When the next segment runs, each layer attends to , where is stop-gradient. Positions are encoded relatively: the attention score between and depends on the distance , through a sinusoid and two learned global bias vectors and . It was introduced by Dai, Yang, Yang, Carbonell, Le and Salakhutdinov in "Transformer-XL: Attentive Language Models Beyond a Fixed-Length Context" (ACL 2019).
A vanilla transformer language model can only be trained on fixed-length chunks. Al-Rfou et al. (2018) trained a 64-layer character model this way: split the corpus into segments, train on each segment independently. That has two costs. No information flows across a segment boundary, so the longest dependency the model can learn is the segment length. And the first tokens of every segment are predicted with almost no context, which the authors call context fragmentation. At evaluation the vanilla model slides its window one token at a time and recomputes the whole segment for every prediction, which is very slow.
Transformer-XL fixes both problems with the recurrence idea from RNNs, applied per layer instead of per token. On release it set new results on all five standard language-modelling benchmarks (abstract): 0.99 bits per character on enwik8, 1.08 on text8, perplexity 18.3 on WikiText-103, 21.8 on One Billion Word and 54.5 on Penn Treebank without fine-tuning. The paper reports dependencies about 80% longer than RNNs and 450% longer than vanilla transformers, and evaluation up to 1,800+ times faster than the vanilla model. XLNet (Yang et al., 2019) was built on it.
Segment-level recurrence
- layer- hidden states of segment ( tokens)
- concatenation along the time axis: key/value positions, query positions
- stop-gradient: the memory is a constant, so backpropagation stays inside the current segment, like truncated BPTT (see Truncated backpropagation through time (TBPTT))
- memory length per layer; the paper sets equal to the segment length in training and makes it several times longer at evaluation
- causal attention over memory + current prefix, then the position-wise MLP, with residuals and LayerNorm
Queries come only from the current segment. Keys and values come from the memory and the current segment, with the usual causal mask inside the segment. The cache is per layer: layer reads the memory of layer , not its own. So the dependency grows by about one segment per layer: a state in the memory already summarises the segment before it. With layers the longest possible dependency is , against for the vanilla model. Compare an RNN, whose hidden state is shifted in time within one layer; Transformer-XL shifts it one layer down.
Relative positional encoding
Recurrence breaks absolute positions. If each segment adds the same embeddings , token 3 of segment and token 3 of segment get identical position encodings even though they are apart, and both sit in the same attention window. The fix is to inject only relative distance, inside the score, in every layer (building on Shaw et al., 2018). Expanding the absolute score gives four terms. Transformer-XL rewrites each one:
- a sinusoid encoding of the distance (no parameters), replacing the absolute
- learned vectors replacing : the query's own position should not matter, so its contribution is the same for every
- separate key projections for content and for position
- in layer the content is the hidden state , not the embedding
Terms (b) and (d) need for every pair, but only distinct distances exist. The paper computes once for , multiplies by the queries to get an matrix indexed by distance, and then shifts row left by to turn it into the matrix indexed by key position. This keeps the positional part linear in the number of distances instead of quadratic. The result is a model whose attention depends on how far back a token is, not on where a segment started, so the memory can be made longer at evaluation than in training.
Transformer-XL at a glance
- Authors / venue
- Dai, Yang, Yang, Carbonell, Le, Salakhutdinov (Carnegie Mellon University and Google Brain), ACL 2019
- Recurrence
- per-layer cache of the previous hidden states, stop-gradient
- Position
- relative sinusoid in every layer, learned , , separate
- Longest dependency
- for layers and segment length
- Results (abstract)
- enwik8 0.99 bpc, text8 1.08 bpc, WikiText-103 18.3 ppl, One Billion Word 21.8 ppl, Penn Treebank 54.5 ppl
- Evaluation speed
- up to 1,800+ times faster than a vanilla transformer that recomputes its window per token
- Descendants
- XLNet (2019), the Compressive Transformer (2019), relative-position schemes in later models
Worked example: one relative score, and how far the context reaches
Take a head with . For query and a key three tokens back (): , , , , .
The four terms of A_rel
- 1(a) content: .
- 2(b) content-dependent position: .
- 3(c) global content bias: . It favours keys with this content whatever the query is.
- 4(d) global position bias: . It favours distance 3 whatever the content is.
- 5. Move the same pair to the next segment: the distance is still 3, so the score is still 1.65. With absolute that repeat per segment, a key tokens further back would get the same position terms as this one.
| layers N | segment n | memory M | vanilla | Transformer-XL | states cached in its cone |
|---|---|---|---|---|---|
| 4 | 8 | 0 | 7 | 7 | 0 |
| 4 | 8 | 1 | 7 | 32 | 27 |
| 4 | 8 | 8 | 7 | 39 | 48 |
| 2 | 8 | 8 | 7 | 23 | 8 |
| 4 | 6 | 6 | 5 | 29 | 36 |
| 6 | 6 | 6 | 5 | 41 | 90 |
With the reach is plus the query's offset in its segment (). Even gives , because information hops through the last state of each earlier segment. A longer memory does not increase the hop count much, but it lets each layer read those tokens directly rather than through a compressed summary. That is why the paper uses longer memories at evaluation time.
1import torch, torch.nn as nn, torch.nn.functional as F23def sinusoid(klen, d):4 # R_k for distances k = klen-1, ..., 1, 0 (descending, as in the paper's code)5 pos = torch.arange(klen - 1, -1, -1.0)6 inv = 1 / (10000 ** (torch.arange(0, d, 2.0) / d))7 ang = pos[:, None] * inv[None, :]8 return torch.cat([ang.sin(), ang.cos()], -1) # (klen, d)910def rel_shift(x):11 # x (B, H, n, klen) indexed by distance; returns scores indexed by key position12 B, H, n, klen = x.shape13 x = F.pad(x, (1, 0)) # (B, H, n, klen+1)14 x = x.view(B, H, klen + 1, n)[:, :, 1:].reshape(B, H, n, klen)15 return x1617class RelAttention(nn.Module):18 def __init__(self, d, H):19 super().__init__()20 self.H, self.dh = H, d // H21 self.qkv = nn.Linear(d, 3 * d, bias=False)22 self.r = nn.Linear(d, d, bias=False) # W_{k,R}23 self.u = nn.Parameter(torch.zeros(H, self.dh)) # global content bias24 self.v = nn.Parameter(torch.zeros(H, self.dh)) # global position bias25 self.o = nn.Linear(d, d, bias=False)26 self.ln = nn.LayerNorm(d)2728 def forward(self, h, mem):29 # h (B, n, d) current segment, mem (B, M, d) cached states of the layer below30 B, n, d = h.shape31 cat = torch.cat([mem.detach(), h], 1) # SG(memory) ∘ h32 klen = cat.size(1)33 q = self.qkv(h)[..., :d] # queries: current segment only34 k, v = self.qkv(cat)[..., d:].chunk(2, -1) # keys/values: memory + segment35 q, k, v = (t.view(B, -1, self.H, self.dh).transpose(1, 2) for t in (q, k, v))36 R = self.r(sinusoid(klen, d).to(h)).view(klen, self.H, self.dh).transpose(0, 1) # (H, klen, dh)37 ac = (q + self.u[:, None]) @ k.transpose(-1, -2) # terms (a) + (c)38 bd = rel_shift((q + self.v[:, None]) @ R.transpose(-1, -2)) # terms (b) + (d)39 score = (ac + bd) / self.dh ** 0.5 # (B, H, n, klen)40 M = klen - n41 causal = torch.ones(n, klen, dtype=torch.bool).triu(M + 1) # query i sees keys <= M + i42 a = score.masked_fill(causal, float("-inf")).softmax(-1) @ v # (B, H, n, dh)43 return self.ln(h + self.o(a.transpose(1, 2).reshape(B, n, d)))4445d, H, n, M, N = 64, 4, 6, 6, 346layers = nn.ModuleList(RelAttention(d, H) for _ in range(N)) # (MLPs omitted for brevity)47mems = [torch.zeros(2, 0, d) for _ in range(N)] # empty memory per layer input48for segment in torch.randn(4, 2, n, d): # 4 consecutive segments, batch 249 h, new_mems = segment, []50 for layer, mem in zip(layers, mems):51 new_mems.append(torch.cat([mem, h], 1)[:, -M:].detach()) # cache this layer's input52 h = layer(h, mem)53 mems = new_mems54print(h.shape, mems[0].shape) # torch.Size([2, 6, 64]) torch.Size([2, 6, 64])Use when
- +Context grows with depth: about tokens instead of
- +No context fragmentation: the first tokens of a segment still see the previous one
- +Fast evaluation: each new token reuses the cache instead of recomputing a window
- +Relative positions generalise to memories longer than in training
Avoid when
- −Gradients never cross a segment boundary, so long-range credit assignment is truncated
- −The cache costs N·M·d floats per sequence and must be carried between batches in order
- −Training data must be fed as contiguous streams, not shuffled independent chunks
- −The relative-attention kernel with the shift trick is more complex than standard attention
Check yourself
Why does Transformer-XL apply stop-gradient to the cached memory?
Check yourself
A Transformer-XL model with segment length n = 4 used per-segment absolute position embeddings instead of relative ones. What goes wrong?
Remember
- ✓Recurrence: ; queries from the segment, keys and values from memory + segment; reach .
- ✓Relative score : content, content-position, global content bias, global position bias.
- ✓Stop-gradient keeps training cost per segment constant, like truncated BPTT; the cache is per layer input.
- ✓Dai et al., ACL 2019: enwik8 0.99 bpc, WikiText-103 18.3 ppl, up to 1,800+× faster evaluation than a vanilla sliding window.
- ✓The shift trick computes the positional terms from M + n distance vectors instead of one per pair.