# Bidirectional LSTM

> A forward and a backward LSTM over the same sequence, concatenated per step. It was the default sequence encoder from 2005 to 2018: phoneme classification, BiLSTM-CRF taggers, ELMo. On padded batches it must be packed, or the backward cell starts on the padding.

Source: https://www.mlfoundry.in/learn/bidirectional-lstm  
Section: Deep Learning › Recurrent Neural Networks and Sequence Models  
Level: core · 10 min read · updated 2026-10-12

**Bidirectional LSTM:** (BiLSTM, BLSTM) is a **Bidirectional RNN** whose two chains are **Long short-term memory (LSTM)** cells: $\overrightarrow h_t,\overrightarrow c_t=\mathrm{LSTM}_{\rightarrow}(x_t,\overrightarrow h_{t-1},\overrightarrow c_{t-1})$, $\overleftarrow h_t,\overleftarrow c_t=\mathrm{LSTM}_{\leftarrow}(x_t,\overleftarrow h_{t+1},\overleftarrow c_{t+1})$, and the output is $y_t=[\overrightarrow h_t;\overleftarrow h_t]\in\mathbb R^{2H}$. The two directions have separate weights and separate cell states.

The bidirectional idea gives each output access to both sides of the sequence; the LSTM cell lets each side carry information over many steps without the gradient vanishing as fast as in a tanh RNN. Together they made the BiLSTM the standard **encoder** for any task where the whole input is available: tagging, offline speech recognition, and the source side of neural machine translation. Everything in **Bidirectional RNN** about what each half can see, and why it cannot stream or generate, applies unchanged; this article is about the LSTM version, where it was used, and how to batch it correctly.

## History: three landmark uses

**Speech (2005).** Graves and Schmidhuber ("Framewise phoneme classification with bidirectional LSTM and other neural network architectures", *Neural Networks*, 2005) classified every 10 ms frame of TIMIT into a phoneme. They found that bidirectional networks beat unidirectional ones and that LSTM was both more accurate and much faster to train than standard RNNs and time-windowed MLPs. Deep BiLSTMs trained with CTC or a transducer later reached a 17.7% phoneme error rate on TIMIT (Graves, Mohamed and Hinton, "Speech recognition with deep recurrent neural networks", ICASSP 2013, abstract). See **Connectionist Temporal Classification** for the loss that removed the need for frame-level alignments.

**Tagging (2015).** Huang, Xu and Yu ("Bidirectional LSTM-CRF models for sequence tagging", 2015) put a linear-chain CRF on top of a BiLSTM for POS tagging, chunking and named-entity recognition. The BiLSTM supplies per-token **emission** scores from both contexts; the CRF adds **transition** scores between neighbouring tags so that, for example, `I-PER` cannot follow `B-LOC`. Lample et al. (2016) and Ma and Hovy (2016) added character-level BiLSTM or CNN features, and BiLSTM-CRF stayed the NER baseline until pretrained Transformers.

**Contextual embeddings (2018).** ELMo (Peters et al., "Deep contextualized word representations", NAACL 2018) is a stack of two biLSTM layers with 4096 units and 512-dimensional projections, a residual connection between the layers, and a character CNN as the input layer, trained as a language model on the 1B Word Benchmark. Crucially it is **not** one bidirectional LM: a forward LM predicts $x_{t+1}$ from the left and a separate backward LM predicts $x_{t-1}$ from the right, sharing only the token representation and softmax, because a bidirectional state would see the word it is asked to predict. A downstream task uses a learned weighted sum of the layers. Adding ELMo raised SQuAD F1 from 81.1 to 85.8 and CoNLL-2003 NER F1 from 90.15 to 92.22 over strong baselines (Table 1). BERT replaced this a few months later with masked-language-model Transformers (see **Self-Attention**).

## Equations and output head

$$
y_t=\big[\overrightarrow h_t;\overleftarrow h_t\big],\qquad P_t=W_e\,y_t+b_e\in\mathbb R^{K},\qquad s(x,\mathbf y)=\sum_{t=1}^{T}\big(A_{y_{t-1},y_t}+P_{t,y_t}\big),\qquad p(\mathbf y\mid x)=\frac{e^{s(x,\mathbf y)}}{\sum_{\mathbf y'}e^{s(x,\mathbf y')}}
$$

- $\overrightarrow h_t,\overleftarrow h_t$: LSTM hidden states of the two directions (gates $i,f,g,o$ and cell $c_t$ as in **Long short-term memory (LSTM)**)
- $P_t$: emission scores for the $K$ tags at token $t$; a plain BiLSTM tagger applies a softmax to $P_t$ and is done
- $A$: $K\times K$ (plus start/stop) transition scores of the CRF layer, learned jointly
- $s(x,\mathbf y)$: score of a whole tag sequence; the partition sum uses the forward algorithm and decoding uses Viterbi, both $O(TK^2)$

**At a glance**

- **Parameters:** per direction $4H(D+H)+8H$ in PyTorch (two bias vectors); $\times2$ for both directions
- **Example:** `nn.LSTM(100, 128, bidirectional=True)`: $2\times117{,}760=235{,}520$ weights; a second layer reads $2H=256$ inputs: $2\times197{,}632$
- **Output:** $(N,T,2H)$; `h_n`, `c_n`: (layers·2, N, H)
- **Typical sizes:** 1–2 layers of 128–512 units per direction for taggers; ELMo 2 × 4096 with 512-d projections
- **Superseded by:** Transformer encoders (BERT and successors) for most NLP; still common in small, offline speech and time-series models

## Padded batches: why packing matters

Sequences in a batch have different lengths, so they are padded to the longest one. A forward LSTM is not hurt by trailing pads at the real positions: it reaches the last real token before it sees any padding, so $\overrightarrow h_t$ for $t\le\ell$ is correct (only the *final* state, taken after the pads, is wrong). The backward LSTM is hurt everywhere: it starts at position $T_{\max}$, chews through $T_{\max}-\ell$ pad vectors, and arrives at the last real token with a nonzero, garbage state. Every backward output of a short sequence is then contaminated, and the error is largest near the end of the real tokens, which is exactly where the backward state should be "fresh".

Pad embeddings of zero do not save you: an LSTM with biases changes its state on a zero input (the forget gate alone multiplies $c$ by $\sigma(b_f)$). The fix is `nn.utils.rnn.pack_padded_sequence`, which stores the batch as a flat sequence of real tokens plus per-step batch sizes. The LSTM then runs each sequence over its true length only, the backward direction starts at token $\ell$ of each sequence, `h_n` holds the forward state at $t=\ell$ and the backward state at $t=1$, and `pad_packed_sequence` puts zeros back at the pad positions. Keras gets the same effect from masking (`mask_zero=True`).

> Figure: A 1-unit BiLSTM (fixed weights, forget bias 1) on three zero-padded sequences of lengths 8, 5 and 2. Packed (default): pads are never read and the backward chain starts at each last real token. Switch to 'padded': the forward outputs at real tokens do not change at all (worst error 0.000), but every backward output of the short sequences does, by up to 0.340 (sequence 3, t = 2 flips from +0.03 to −0.31). The final states are wrong too: for sequence 3, h_n forward becomes 0.48 instead of 0.24 and h_n backward −0.54 instead of −0.22. Click a cell to change a sequence's length.

## Worked example: the length-2 sequence

Sequence 3 in the figure is $x=(0.7931,\,-0.4238)$ followed by six pads. The backward LSTM uses $W_x=(-0.7,\,0.5,\,-1.2,\,0.9)$, $W_h=(0.6,\,-0.4,\,0.7,\,0.3)$ and $b=(0.1,\,1.0,\,-0.4,\,0.5)$ for the gates $(i,f,g,o)$, starting from $h=c=0$.

**First backward step, packed vs padded (verified in Node)**

1. **Packed** starts at the real token $x_2=-0.4238$: pre-activations $(0.3967,\,0.7881,\,0.1086,\,0.1186)$, so $i=0.5979$, $f=0.6874$, $g=0.1081$, $o=0.5296$. Then $c=i\,g=0.0647$ and $\overleftarrow h_2=o\tanh c=0.0342$.
2. **Padded** starts at a pad $x_8=0$: the pre-activations are just the biases, $i=\sigma(0.1)=0.5250$, $f=\sigma(1)=0.7311$, $g=\tanh(-0.4)=-0.3799$, $o=\sigma(0.5)=0.6225$, so $c=-0.1995$ and $h=-0.1225$. Zero input, nonzero state.
3. After six pads the padded chain reaches the real token with $h=-0.4132$, $c=-0.8568$ instead of $0,0$, and produces $\overleftarrow h_2=-0.3059$: the wrong sign.
4. One more step gives $\overleftarrow h_1=-0.5414$ (padded) against $-0.2178$ (packed). This is `h_n[1]`, the backward half of the sentence vector.

All three sequences of the figure: error of the padded run at real tokens and in h_n

| sequence | length | max \|Δ\| forward outputs | max \|Δ\| backward outputs | \|Δ\| h_n forward | \|Δ\| h_n backward |
| --- | --- | --- | --- | --- | --- |
| 1 | 8 | 0.000 | 0.000 | 0.000 | 0.000 |
| 2 | 5 | 0.000 | 0.214 | 0.039 | 0.066 |
| 3 | 2 | 0.000 | 0.340 | 0.238 | 0.324 |

The full-length sequence is unaffected, so a bug like this hides well: it only hurts short sequences in batches with long ones, and it changes with the batch composition.

```python
import torch, torch.nn as nn
import torch.nn.functional as F
from torch.nn.utils.rnn import pack_padded_sequence, pad_packed_sequence

class BiLSTMTagger(nn.Module):
    def __init__(self, vocab, n_tags, emb=100, hidden=128, layers=2, pad_idx=0, dropout=0.5):
        super().__init__()
        self.emb = nn.Embedding(vocab, emb, padding_idx=pad_idx)
        self.lstm = nn.LSTM(emb, hidden, num_layers=layers, bidirectional=True,
                            batch_first=True, dropout=dropout)
        self.out = nn.Linear(2 * hidden, n_tags)      # emission scores P_t

    def forward(self, tokens, lengths):               # tokens (N, T) padded, lengths (N,)
        x = self.emb(tokens)
        packed = pack_padded_sequence(x, lengths.cpu(), batch_first=True,
                                      enforce_sorted=False)   # lengths must live on the CPU
        out, (h_n, c_n) = self.lstm(packed)
        out, _ = pad_packed_sequence(out, batch_first=True, total_length=tokens.size(1))
        return self.out(out), h_n                     # (N, T, n_tags), (layers*2, N, hidden)

tokens  = torch.tensor([[5, 8, 2, 9, 4],
                        [7, 3, 0, 0, 0]])             # second sentence: 2 real tokens
lengths = torch.tensor([5, 2])
tags    = torch.tensor([[1, 0, 2, 3, 1],
                        [2, 1, -100, -100, -100]])    # -100 = ignored (padding)

model = BiLSTMTagger(vocab=10, n_tags=4)
emissions, h_n = model(tokens, lengths)
print(emissions.shape, h_n.shape)    # torch.Size([2, 5, 4]) torch.Size([4, 2, 128])

loss = F.cross_entropy(emissions.transpose(1, 2), tags, ignore_index=-100)   # (N, C, T) vs (N, T)
loss.backward()

# Sentence vector: last layer, forward state at t = len and backward state at t = 1.
# With packing, h_n is already in the original batch order (enforce_sorted=False un-sorts it).
sent = torch.cat([h_n[-2], h_n[-1]], dim=-1)          # (N, 256)
```

A BiLSTM tagger with packed sequences (add a CRF layer on top of the emissions for BiLSTM-CRF)

> **Packing is not optional for the backward half:** Without `pack_padded_sequence` the code runs and trains, but the backward LSTM reads the padding first. Symptoms: accuracy depends on how batches are bucketed, short sentences do worse than they should, and evaluation with batch size 1 gives different results from batched evaluation. Also remember that `lengths` must be a CPU tensor or list, and that `out[:, -1]` is not a sentence vector (see **Bidirectional RNN**).

> **Why ELMo needed two language models:** A biLSTM state at $t$ has read $x_{t+1}$, so it cannot be trained to predict $x_{t+1}$. ELMo therefore trains a forward LM and a backward LM with separate LSTMs and only concatenates their states **after** training, as features. BERT's masked LM solves the same problem differently: hide the token, then let every layer look both ways.

> **Where BiLSTMs still appear:** Encoders of Bahdanau-style translation models (see **Bahdanau attention**; GNMT made only its first encoder layer bidirectional, see **Stacked RNN**), offline speech and handwriting recognisers trained with CTC, small-footprint NER and slot-filling models, and biological sequence labelling. For streaming they are replaced by unidirectional or latency-controlled variants.

**Strengths**

- Full left and right context with LSTM-quality long-range memory on each side
- Strong, data-efficient encoder for tagging and offline speech; natural emission model for a CRF
- Linear time and memory in sequence length
- Mature tooling: `bidirectional=True`, packed sequences, cuDNN kernels

**Limitations**

- No streaming or autoregressive use
- Sequential in time: much slower to train than a Transformer encoder on long inputs
- Twice the parameters of one direction; $2H$-wide outputs
- Silent bugs with padding and with out[:, -1] unless sequences are packed

**Question:** You feed a zero-padded batch to nn.LSTM(bidirectional=True) without packing. Which outputs at real (non-pad) positions are wrong?

- A. Only the backward outputs of sequences shorter than the batch maximum
- B. Only the forward outputs of short sequences
- C. Both directions at every real position
- D. None, because the pad embedding is zero

**Answer:** A. Only the backward outputs of sequences shorter than the batch maximum — The forward LSTM processes the real tokens before it meets any pad, so its outputs there are exact (the figure's worst error is 0.000); only its final state, read after the pads, is wrong. The backward LSTM starts at the last pad and carries a garbage state into every real token of a short sequence. Zero inputs still move an LSTM's state through its biases.

**Question:** Why does ELMo train a forward LSTM language model and a separate backward one instead of a single bidirectional LSTM LM?

- A. A bidirectional state at t has already read the token it would be asked to predict
- B. Two unidirectional models have fewer parameters
- C. cuDNN does not support bidirectional LSTMs
- D. The backward LM is only used at inference time

**Answer:** A. A bidirectional state at t has already read the token it would be asked to predict — The backward half at $t$ summarises $x_t,\dots,x_T$, which includes $x_{t+1}$, so a joint next-token objective would be trivial. Each ELMo LM predicts only from its own side, and the two sets of states are concatenated afterwards as contextual features.

**Key points**

- BiLSTM: forward and backward LSTMs with separate weights and cell states, output $y_t=[\overrightarrow h_t;\overleftarrow h_t]$ of size $2H$.
- Landmarks: framewise phoneme classification (Graves & Schmidhuber, 2005), BiLSTM-CRF tagging (Huang et al., 2015), ELMo's two-layer biLM (Peters et al., 2018).
- BiLSTM-CRF: the BiLSTM gives emission scores $P_t$, the CRF adds tag-transition scores $A$; Viterbi decoding is $O(TK^2)$.
- Pack padded batches: otherwise the backward LSTM starts on the pads and corrupts every backward output of short sequences, plus both halves of `h_n`.
- Parameters per direction $4H(D+H)+8H$ in PyTorch; `nn.LSTM(100, 128, bidirectional=True)` has 235,520.

---

From MLFoundry (https://www.mlfoundry.in). The web page adds interactive figures and diagrams.
