Definition
Bidirectional LSTM (BiLSTM, BLSTM) is a Bidirectional RNN whose two chains are Long short-term memory (LSTM) cells: , , and the output is . The two directions have separate weights and separate cell states.
The bidirectional idea gives each output access to both sides of the sequence; the LSTM cell lets each side carry information over many steps without the gradient vanishing as fast as in a tanh RNN. Together they made the BiLSTM the standard encoder for any task where the whole input is available: tagging, offline speech recognition, and the source side of neural machine translation. Everything in Bidirectional RNN about what each half can see, and why it cannot stream or generate, applies unchanged; this article is about the LSTM version, where it was used, and how to batch it correctly.
History: three landmark uses
Speech (2005). Graves and Schmidhuber ("Framewise phoneme classification with bidirectional LSTM and other neural network architectures", Neural Networks, 2005) classified every 10 ms frame of TIMIT into a phoneme. They found that bidirectional networks beat unidirectional ones and that LSTM was both more accurate and much faster to train than standard RNNs and time-windowed MLPs. Deep BiLSTMs trained with CTC or a transducer later reached a 17.7% phoneme error rate on TIMIT (Graves, Mohamed and Hinton, "Speech recognition with deep recurrent neural networks", ICASSP 2013, abstract). See Connectionist Temporal Classification for the loss that removed the need for frame-level alignments.
Tagging (2015). Huang, Xu and Yu ("Bidirectional LSTM-CRF models for sequence tagging", 2015) put a linear-chain CRF on top of a BiLSTM for POS tagging, chunking and named-entity recognition. The BiLSTM supplies per-token emission scores from both contexts; the CRF adds transition scores between neighbouring tags so that, for example, I-PER cannot follow B-LOC. Lample et al. (2016) and Ma and Hovy (2016) added character-level BiLSTM or CNN features, and BiLSTM-CRF stayed the NER baseline until pretrained Transformers.
Contextual embeddings (2018). ELMo (Peters et al., "Deep contextualized word representations", NAACL 2018) is a stack of two biLSTM layers with 4096 units and 512-dimensional projections, a residual connection between the layers, and a character CNN as the input layer, trained as a language model on the 1B Word Benchmark. Crucially it is not one bidirectional LM: a forward LM predicts from the left and a separate backward LM predicts from the right, sharing only the token representation and softmax, because a bidirectional state would see the word it is asked to predict. A downstream task uses a learned weighted sum of the layers. Adding ELMo raised SQuAD F1 from 81.1 to 85.8 and CoNLL-2003 NER F1 from 90.15 to 92.22 over strong baselines (Table 1). BERT replaced this a few months later with masked-language-model Transformers (see Self-Attention).
Equations and output head
- LSTM hidden states of the two directions (gates and cell as in Long short-term memory (LSTM))
- emission scores for the tags at token ; a plain BiLSTM tagger applies a softmax to and is done
- (plus start/stop) transition scores of the CRF layer, learned jointly
- score of a whole tag sequence; the partition sum uses the forward algorithm and decoding uses Viterbi, both
At a glance
- Parameters
- per direction in PyTorch (two bias vectors); for both directions
- Example
nn.LSTM(100, 128, bidirectional=True): weights; a second layer reads inputs:- Output
- ;
h_n,c_n: (layers·2, N, H) - Typical sizes
- 1–2 layers of 128–512 units per direction for taggers; ELMo 2 × 4096 with 512-d projections
- Superseded by
- Transformer encoders (BERT and successors) for most NLP; still common in small, offline speech and time-series models
Padded batches: why packing matters
Sequences in a batch have different lengths, so they are padded to the longest one. A forward LSTM is not hurt by trailing pads at the real positions: it reaches the last real token before it sees any padding, so for is correct (only the final state, taken after the pads, is wrong). The backward LSTM is hurt everywhere: it starts at position , chews through pad vectors, and arrives at the last real token with a nonzero, garbage state. Every backward output of a short sequence is then contaminated, and the error is largest near the end of the real tokens, which is exactly where the backward state should be "fresh".
Pad embeddings of zero do not save you: an LSTM with biases changes its state on a zero input (the forget gate alone multiplies by ). The fix is nn.utils.rnn.pack_padded_sequence, which stores the batch as a flat sequence of real tokens plus per-step batch sizes. The LSTM then runs each sequence over its true length only, the backward direction starts at token of each sequence, h_n holds the forward state at and the backward state at , and pad_packed_sequence puts zeros back at the pad positions. Keras gets the same effect from masking (mask_zero=True).
Worked example: the length-2 sequence
Sequence 3 in the figure is followed by six pads. The backward LSTM uses , and for the gates , starting from .
First backward step, packed vs padded (verified in Node)
- 1Packed starts at the real token : pre-activations , so , , , . Then and .
- 2Padded starts at a pad : the pre-activations are just the biases, , , , , so and . Zero input, nonzero state.
- 3After six pads the padded chain reaches the real token with , instead of , and produces : the wrong sign.
- 4One more step gives (padded) against (packed). This is
h_n[1], the backward half of the sentence vector.
| sequence | length | max |Δ| forward outputs | max |Δ| backward outputs | |Δ| h_n forward | |Δ| h_n backward |
|---|---|---|---|---|---|
| 1 | 8 | 0.000 | 0.000 | 0.000 | 0.000 |
| 2 | 5 | 0.000 | 0.214 | 0.039 | 0.066 |
| 3 | 2 | 0.000 | 0.340 | 0.238 | 0.324 |
The full-length sequence is unaffected, so a bug like this hides well: it only hurts short sequences in batches with long ones, and it changes with the batch composition.
1import torch, torch.nn as nn2import torch.nn.functional as F3from torch.nn.utils.rnn import pack_padded_sequence, pad_packed_sequence45class BiLSTMTagger(nn.Module):6 def __init__(self, vocab, n_tags, emb=100, hidden=128, layers=2, pad_idx=0, dropout=0.5):7 super().__init__()8 self.emb = nn.Embedding(vocab, emb, padding_idx=pad_idx)9 self.lstm = nn.LSTM(emb, hidden, num_layers=layers, bidirectional=True,10 batch_first=True, dropout=dropout)11 self.out = nn.Linear(2 * hidden, n_tags) # emission scores P_t1213 def forward(self, tokens, lengths): # tokens (N, T) padded, lengths (N,)14 x = self.emb(tokens)15 packed = pack_padded_sequence(x, lengths.cpu(), batch_first=True,16 enforce_sorted=False) # lengths must live on the CPU17 out, (h_n, c_n) = self.lstm(packed)18 out, _ = pad_packed_sequence(out, batch_first=True, total_length=tokens.size(1))19 return self.out(out), h_n # (N, T, n_tags), (layers*2, N, hidden)2021tokens = torch.tensor([[5, 8, 2, 9, 4],22 [7, 3, 0, 0, 0]]) # second sentence: 2 real tokens23lengths = torch.tensor([5, 2])24tags = torch.tensor([[1, 0, 2, 3, 1],25 [2, 1, -100, -100, -100]]) # -100 = ignored (padding)2627model = BiLSTMTagger(vocab=10, n_tags=4)28emissions, h_n = model(tokens, lengths)29print(emissions.shape, h_n.shape) # torch.Size([2, 5, 4]) torch.Size([4, 2, 128])3031loss = F.cross_entropy(emissions.transpose(1, 2), tags, ignore_index=-100) # (N, C, T) vs (N, T)32loss.backward()3334# Sentence vector: last layer, forward state at t = len and backward state at t = 1.35# With packing, h_n is already in the original batch order (enforce_sorted=False un-sorts it).36sent = torch.cat([h_n[-2], h_n[-1]], dim=-1) # (N, 256)Use when
- +Full left and right context with LSTM-quality long-range memory on each side
- +Strong, data-efficient encoder for tagging and offline speech; natural emission model for a CRF
- +Linear time and memory in sequence length
- +Mature tooling:
bidirectional=True, packed sequences, cuDNN kernels
Avoid when
- −No streaming or autoregressive use
- −Sequential in time: much slower to train than a Transformer encoder on long inputs
- −Twice the parameters of one direction; -wide outputs
- −Silent bugs with padding and with out[:, -1] unless sequences are packed
Check yourself
You feed a zero-padded batch to nn.LSTM(bidirectional=True) without packing. Which outputs at real (non-pad) positions are wrong?
Check yourself
Why does ELMo train a forward LSTM language model and a separate backward one instead of a single bidirectional LSTM LM?
Remember
- ✓BiLSTM: forward and backward LSTMs with separate weights and cell states, output of size .
- ✓Landmarks: framewise phoneme classification (Graves & Schmidhuber, 2005), BiLSTM-CRF tagging (Huang et al., 2015), ELMo's two-layer biLM (Peters et al., 2018).
- ✓BiLSTM-CRF: the BiLSTM gives emission scores , the CRF adds tag-transition scores ; Viterbi decoding is .
- ✓Pack padded batches: otherwise the backward LSTM starts on the pads and corrupts every backward output of short sequences, plus both halves of
h_n. - ✓Parameters per direction in PyTorch;
nn.LSTM(100, 128, bidirectional=True)has 235,520.