Bidirectional LSTM

A forward and a backward LSTM over the same sequence, concatenated per step. It was the default sequence encoder from 2005 to 2018: phoneme classification, BiLSTM-CRF taggers, ELMo. On padded batches it must be packed, or the backward cell starts on the padding.

10 min read1074 wordscore0 viewsupdated 2026-10-12

Definition

Bidirectional LSTM (BiLSTM, BLSTM) is a Bidirectional RNN whose two chains are Long short-term memory (LSTM) cells: h→t,c→t=LSTM→(xt,h→t−1,c→t−1)\overrightarrow h_t,\overrightarrow c_t=\mathrm{LSTM}_{\rightarrow}(x_t,\overrightarrow h_{t-1},\overrightarrow c_{t-1}), h←t,c←t=LSTM←(xt,h←t+1,c←t+1)\overleftarrow h_t,\overleftarrow c_t=\mathrm{LSTM}_{\leftarrow}(x_t,\overleftarrow h_{t+1},\overleftarrow c_{t+1}), and the output is yt=[h→t;h←t]∈R2Hy_t=[\overrightarrow h_t;\overleftarrow h_t]\in\mathbb R^{2H}. The two directions have separate weights and separate cell states.

The bidirectional idea gives each output access to both sides of the sequence; the LSTM cell lets each side carry information over many steps without the gradient vanishing as fast as in a tanh RNN. Together they made the BiLSTM the standard encoder for any task where the whole input is available: tagging, offline speech recognition, and the source side of neural machine translation. Everything in Bidirectional RNN about what each half can see, and why it cannot stream or generate, applies unchanged; this article is about the LSTM version, where it was used, and how to batch it correctly.

History: three landmark uses

Speech (2005). Graves and Schmidhuber ("Framewise phoneme classification with bidirectional LSTM and other neural network architectures", Neural Networks, 2005) classified every 10 ms frame of TIMIT into a phoneme. They found that bidirectional networks beat unidirectional ones and that LSTM was both more accurate and much faster to train than standard RNNs and time-windowed MLPs. Deep BiLSTMs trained with CTC or a transducer later reached a 17.7% phoneme error rate on TIMIT (Graves, Mohamed and Hinton, "Speech recognition with deep recurrent neural networks", ICASSP 2013, abstract). See Connectionist Temporal Classification for the loss that removed the need for frame-level alignments.

Tagging (2015). Huang, Xu and Yu ("Bidirectional LSTM-CRF models for sequence tagging", 2015) put a linear-chain CRF on top of a BiLSTM for POS tagging, chunking and named-entity recognition. The BiLSTM supplies per-token emission scores from both contexts; the CRF adds transition scores between neighbouring tags so that, for example, I-PER cannot follow B-LOC. Lample et al. (2016) and Ma and Hovy (2016) added character-level BiLSTM or CNN features, and BiLSTM-CRF stayed the NER baseline until pretrained Transformers.

Contextual embeddings (2018). ELMo (Peters et al., "Deep contextualized word representations", NAACL 2018) is a stack of two biLSTM layers with 4096 units and 512-dimensional projections, a residual connection between the layers, and a character CNN as the input layer, trained as a language model on the 1B Word Benchmark. Crucially it is not one bidirectional LM: a forward LM predicts xt+1x_{t+1} from the left and a separate backward LM predicts xt−1x_{t-1} from the right, sharing only the token representation and softmax, because a bidirectional state would see the word it is asked to predict. A downstream task uses a learned weighted sum of the layers. Adding ELMo raised SQuAD F1 from 81.1 to 85.8 and CoNLL-2003 NER F1 from 90.15 to 92.22 over strong baselines (Table 1). BERT replaced this a few months later with masked-language-model Transformers (see Self-Attention).

Equations and output head

yt=[h→t;h←t],Pt=We yt+be∈RK,s(x,y)=∑t=1T(Ayt−1,yt+Pt,yt),p(y∣x)=es(x,y)∑y′es(x,y′)y_t=\big[\overrightarrow h_t;\overleftarrow h_t\big],\qquad P_t=W_e\,y_t+b_e\in\mathbb R^{K},\qquad s(x,\mathbf y)=\sum_{t=1}^{T}\big(A_{y_{t-1},y_t}+P_{t,y_t}\big),\qquad p(\mathbf y\mid x)=\frac{e^{s(x,\mathbf y)}}{\sum_{\mathbf y'}e^{s(x,\mathbf y')}}
h→t,h←t\overrightarrow h_t,\overleftarrow h_t
LSTM hidden states of the two directions (gates i,f,g,oi,f,g,o and cell ctc_t as in Long short-term memory (LSTM))
PtP_t
emission scores for the KK tags at token tt; a plain BiLSTM tagger applies a softmax to PtP_t and is done
AA
K×KK\times K (plus start/stop) transition scores of the CRF layer, learned jointly
s(x,y)s(x,\mathbf y)
score of a whole tag sequence; the partition sum uses the forward algorithm and decoding uses Viterbi, both O(TK2)O(TK^2)

At a glance

Parameters
per direction 4H(D+H)+8H4H(D+H)+8H in PyTorch (two bias vectors); ×2\times2 for both directions
Example
nn.LSTM(100, 128, bidirectional=True): 2×117,760=235,5202\times117{,}760=235{,}520 weights; a second layer reads 2H=2562H=256 inputs: 2×197,6322\times197{,}632
Output
(N,T,2H)(N,T,2H); h_n, c_n: (layers·2, N, H)
Typical sizes
1–2 layers of 128–512 units per direction for taggers; ELMo 2 × 4096 with 512-d projections
Superseded by
Transformer encoders (BERT and successors) for most NLP; still common in small, offline speech and time-series models

Padded batches: why packing matters

Sequences in a batch have different lengths, so they are padded to the longest one. A forward LSTM is not hurt by trailing pads at the real positions: it reaches the last real token before it sees any padding, so h→t\overrightarrow h_t for t≤ℓt\le\ell is correct (only the final state, taken after the pads, is wrong). The backward LSTM is hurt everywhere: it starts at position Tmax⁡T_{\max}, chews through Tmax⁡−ℓT_{\max}-\ell pad vectors, and arrives at the last real token with a nonzero, garbage state. Every backward output of a short sequence is then contaminated, and the error is largest near the end of the real tokens, which is exactly where the backward state should be "fresh".

Pad embeddings of zero do not save you: an LSTM with biases changes its state on a zero input (the forget gate alone multiplies cc by σ(bf)\sigma(b_f)). The fix is nn.utils.rnn.pack_padded_sequence, which stores the batch as a flat sequence of real tokens plus per-step batch sizes. The LSTM then runs each sequence over its true length only, the backward direction starts at token ℓ\ell of each sequence, h_n holds the forward state at t=ℓt=\ell and the backward state at t=1t=1, and pad_packed_sequence puts zeros back at the pad positions. Keras gets the same effect from masking (mask_zero=True).

1-unit BiLSTM on a zero-padded batch, packed (pack_padded_sequence)output[:, t, :H] forwardseq 10.270.480.280.460.230.410.530.46start→seq 20.410.200.480.550.55padpadpadstart→seq 30.400.24padpadpadpadpadpadstart→output[:, t, H:] backwardseq 1−0.44−0.37−0.14−0.30−0.10−0.37−0.30−0.09←startseq 2−0.51−0.25−0.47−0.37−0.20padpadpad←startseq 3−0.220.03padpadpadpadpadpad←startt=1t=2t=3t=4t=5t=6t=7t=8final states h_npackedpaddedh_n[0] = forward at t = lenseq 1 (len 8)0.460.46okseq 2 (len 5)0.550.51Δ0.04seq 3 (len 2)0.240.48Δ0.24h_n[1] = backward at t = 1seq 1 (len 8)−0.44−0.44okseq 2 (len 5)−0.51−0.58Δ0.07seq 3 (len 2)−0.22−0.54Δ0.32padded run, worst real-token error:forward 0.000 backward 0.340forward outputs before the pads areunaffected; backward ones are not.click a cell to set that sequence's length · Δ = error vs packed at a real tokenpacked: the backward LSTM starts at each last real token; pads are never read
lengths 8, 5, 2
fig.A 1-unit BiLSTM (fixed weights, forget bias 1) on three zero-padded sequences of lengths 8, 5 and 2. Packed (default): pads are never read and the backward chain starts at each last real token. Switch to 'padded': the forward outputs at real tokens do not change at all (worst error 0.000), but every backward output of the short sequences does, by up to 0.340 (sequence 3, t = 2 flips from +0.03 to −0.31). The final states are wrong too: for sequence 3, h_n forward becomes 0.48 instead of 0.24 and h_n backward −0.54 instead of −0.22. Click a cell to change a sequence's length.

Worked example: the length-2 sequence

Sequence 3 in the figure is x=(0.7931, −0.4238)x=(0.7931,\,-0.4238) followed by six pads. The backward LSTM uses Wx=(−0.7, 0.5, −1.2, 0.9)W_x=(-0.7,\,0.5,\,-1.2,\,0.9), Wh=(0.6, −0.4, 0.7, 0.3)W_h=(0.6,\,-0.4,\,0.7,\,0.3) and b=(0.1, 1.0, −0.4, 0.5)b=(0.1,\,1.0,\,-0.4,\,0.5) for the gates (i,f,g,o)(i,f,g,o), starting from h=c=0h=c=0.

First backward step, packed vs padded (verified in Node)

  1. 1Packed starts at the real token x2=−0.4238x_2=-0.4238: pre-activations (0.3967, 0.7881, 0.1086, 0.1186)(0.3967,\,0.7881,\,0.1086,\,0.1186), so i=0.5979i=0.5979, f=0.6874f=0.6874, g=0.1081g=0.1081, o=0.5296o=0.5296. Then c=i g=0.0647c=i\,g=0.0647 and h←2=otanh⁡c=0.0342\overleftarrow h_2=o\tanh c=0.0342.
  2. 2Padded starts at a pad x8=0x_8=0: the pre-activations are just the biases, i=σ(0.1)=0.5250i=\sigma(0.1)=0.5250, f=σ(1)=0.7311f=\sigma(1)=0.7311, g=tanh⁡(−0.4)=−0.3799g=\tanh(-0.4)=-0.3799, o=σ(0.5)=0.6225o=\sigma(0.5)=0.6225, so c=−0.1995c=-0.1995 and h=−0.1225h=-0.1225. Zero input, nonzero state.
  3. 3After six pads the padded chain reaches the real token with h=−0.4132h=-0.4132, c=−0.8568c=-0.8568 instead of 0,00,0, and produces h←2=−0.3059\overleftarrow h_2=-0.3059: the wrong sign.
  4. 4One more step gives h←1=−0.5414\overleftarrow h_1=-0.5414 (padded) against −0.2178-0.2178 (packed). This is h_n[1], the backward half of the sentence vector.
sequencelengthmax |Δ| forward outputsmax |Δ| backward outputs|Δ| h_n forward|Δ| h_n backward
180.0000.0000.0000.000
250.0000.2140.0390.066
320.0000.3400.2380.324
All three sequences of the figure: error of the padded run at real tokens and in h_n

The full-length sequence is unaffected, so a bug like this hides well: it only hurts short sequences in batches with long ones, and it changes with the batch composition.

python
1import torch, torch.nn as nn
2import torch.nn.functional as F
3from torch.nn.utils.rnn import pack_padded_sequence, pad_packed_sequence
4
5class BiLSTMTagger(nn.Module):
6 def __init__(self, vocab, n_tags, emb=100, hidden=128, layers=2, pad_idx=0, dropout=0.5):
7 super().__init__()
8 self.emb = nn.Embedding(vocab, emb, padding_idx=pad_idx)
9 self.lstm = nn.LSTM(emb, hidden, num_layers=layers, bidirectional=True,
10 batch_first=True, dropout=dropout)
11 self.out = nn.Linear(2 * hidden, n_tags) # emission scores P_t
12
13 def forward(self, tokens, lengths): # tokens (N, T) padded, lengths (N,)
14 x = self.emb(tokens)
15 packed = pack_padded_sequence(x, lengths.cpu(), batch_first=True,
16 enforce_sorted=False) # lengths must live on the CPU
17 out, (h_n, c_n) = self.lstm(packed)
18 out, _ = pad_packed_sequence(out, batch_first=True, total_length=tokens.size(1))
19 return self.out(out), h_n # (N, T, n_tags), (layers*2, N, hidden)
20
21tokens = torch.tensor([[5, 8, 2, 9, 4],
22 [7, 3, 0, 0, 0]]) # second sentence: 2 real tokens
23lengths = torch.tensor([5, 2])
24tags = torch.tensor([[1, 0, 2, 3, 1],
25 [2, 1, -100, -100, -100]]) # -100 = ignored (padding)
26
27model = BiLSTMTagger(vocab=10, n_tags=4)
28emissions, h_n = model(tokens, lengths)
29print(emissions.shape, h_n.shape) # torch.Size([2, 5, 4]) torch.Size([4, 2, 128])
30
31loss = F.cross_entropy(emissions.transpose(1, 2), tags, ignore_index=-100) # (N, C, T) vs (N, T)
32loss.backward()
33
34# Sentence vector: last layer, forward state at t = len and backward state at t = 1.
35# With packing, h_n is already in the original batch order (enforce_sorted=False un-sorts it).
36sent = torch.cat([h_n[-2], h_n[-1]], dim=-1) # (N, 256)
A BiLSTM tagger with packed sequences (add a CRF layer on top of the emissions for BiLSTM-CRF)

Use when

  • +Full left and right context with LSTM-quality long-range memory on each side
  • +Strong, data-efficient encoder for tagging and offline speech; natural emission model for a CRF
  • +Linear time and memory in sequence length
  • +Mature tooling: bidirectional=True, packed sequences, cuDNN kernels

Avoid when

  • −No streaming or autoregressive use
  • −Sequential in time: much slower to train than a Transformer encoder on long inputs
  • −Twice the parameters of one direction; 2H2H-wide outputs
  • −Silent bugs with padding and with out[:, -1] unless sequences are packed

Check yourself

You feed a zero-padded batch to nn.LSTM(bidirectional=True) without packing. Which outputs at real (non-pad) positions are wrong?

Check yourself

Why does ELMo train a forward LSTM language model and a separate backward one instead of a single bidirectional LSTM LM?

Remember

  • ✓BiLSTM: forward and backward LSTMs with separate weights and cell states, output yt=[h→t;h←t]y_t=[\overrightarrow h_t;\overleftarrow h_t] of size 2H2H.
  • ✓Landmarks: framewise phoneme classification (Graves & Schmidhuber, 2005), BiLSTM-CRF tagging (Huang et al., 2015), ELMo's two-layer biLM (Peters et al., 2018).
  • ✓BiLSTM-CRF: the BiLSTM gives emission scores PtP_t, the CRF adds tag-transition scores AA; Viterbi decoding is O(TK2)O(TK^2).
  • ✓Pack padded batches: otherwise the backward LSTM starts on the pads and corrupts every backward output of short sequences, plus both halves of h_n.
  • ✓Parameters per direction 4H(D+H)+8H4H(D+H)+8H in PyTorch; nn.LSTM(100, 128, bidirectional=True) has 235,520.
← back to Deep Learning roadmap