Definition
Bidirectional RNN (BRNN) runs two independent recurrent networks over the same input: a forward RNN from to and a backward RNN from down to . The representation at step is the pair , which summarises in the first half and in the second.
A unidirectional Recurrent neural network (RNN) compresses only the past into . That is the right inductive bias for generation, where the future does not exist yet, but it is a handicap for labelling a sequence that is already fully available. To tag the word "bank" as a river bank or a financial bank, or to classify the phoneme at 300 ms of an utterance, the words or frames that come after are as informative as those before.
The obvious fixes were unsatisfying. A time window of frames fed to a feed-forward net fixes the amount of context in advance. Delaying the output of a forward RNN by steps lets it see frames of the future, but the network then has to remember the past for extra steps and must be tuned. Schuster and Paliwal ("Bidirectional recurrent neural networks", IEEE Transactions on Signal Processing, 1997) proposed to split the state into a forward and a backward part with no connections between them, so both can be trained with ordinary backpropagation through time and each output uses all of the input. Baldi, Brunak, Frasconi, Soda and Pollastri (1999) applied the same idea to protein secondary-structure prediction, and Graves and Schmidhuber (2005) combined it with LSTM cells; see Bidirectional LSTM.
Equations
- forward state, a function of ; starts from (zeros by default)
- backward state, a function of ; starts from
- separate weights per direction: nothing is shared between the two chains
- the cell: for a vanilla RNN, or an LSTM / GRU cell
- the concatenation, size ; a task head reads it, e.g. for tagging
Because the two chains never feed into each other, the unrolled graph is still acyclic. Backpropagation through time runs on each chain separately: the gradient of a loss at flows left through the forward chain and right through the backward chain. Concatenation is the usual merge; Keras's Bidirectional wrapper also offers sum, mean and product (merge_mode), which keep the size at .
At a glance
- Context at step t
- all of : past via , future via
- Output size
- per step (concat), with sum/mean merge
- Parameters
- exactly twice a unidirectional layer of the same
- Compute
- two passes of steps; they are independent and can run in parallel
- Latency
- the first output needs the last input: not usable for streaming or for generating text
- Typical uses
- tagging (POS, NER), speech frame classification, encoders for translation and other seq2seq tasks
What each half can see
For one layer, the Jacobian of the forward state with respect to an input is a product of one-step Jacobians, for , and exactly zero for . The backward state mirrors it. So is sensitive to every input, but not symmetrically: each half is most sensitive to inputs near , and the influence of distant inputs decays with the same vanishing-gradient factors as in any RNN (see Vanilla RNN). A bidirectional net doubles the reach of the context, not its quality.
Notice what does not contain: no single state has read the whole sequence in one pass. never saw and never saw . Interactions between far-left and far-right context have to be formed by the layer above, which is one reason bidirectional layers are usually stacked (see Stacked RNN): in layer 2 each direction reads the concatenated states of layer 1 and therefore mixes both contexts.
Worked example: three steps by hand
Take scalar states, , input weight 1, recurrent weight in both directions, zero biases and zero initial states, and the sequence . Values are rounded to 4 decimals and were checked with the figure's library:
| t | xₜ | h→ₜ = tanh(xₜ + 0.5·h→ₜ₋₁) | h←ₜ = tanh(xₜ + 0.5·h←ₜ₊₁) | yₜ |
|---|---|---|---|---|
| 1 | 1 | [0.7616; 0.7244] | ||
| 2 | −0.5 | [−0.1186; −0.1664] | ||
| 3 | 0.8 | [0.6296; 0.6640] |
Sensitivities of y₂, and what streaming would cost
- 1Forward half: , , and .
- 2Backward half: , , and . Finite differences give the same four numbers.
- 3Streaming: suppose only have arrived. The backward chain starts at and gives , .
- 4When arrives, both must be recomputed: moves by and by . Any label emitted early would have been based on wrong states.
When you cannot use it
The backward chain starts at the last input, so a bidirectional layer needs the complete sequence before it can output anything. That rules it out for three common settings:
Where it's used
Language modelling and decoders
Predicting from a state that has already read is cheating: the backward half would simply copy the answer. Decoders in Sequence-to-sequence (Seq2Seq) models and autoregressive LMs must be causal. (ELMo trains a forward and a backward LM separately for exactly this reason; see Bidirectional LSTM.)
Streaming speech and online control
A live captioner or keyword spotter must emit while audio arrives. Production systems use unidirectional cells, a small fixed look-ahead, or chunked "latency-controlled" BLSTMs (Zhang et al., 2016) whose backward pass only covers a short future chunk.
Long streams with stateful training
The backward state at depends on everything up to the end of the stream, so carrying states across chunks (see Truncated backpropagation through time (TBPTT)) only makes sense for the forward direction.
Where the whole input is available, bidirectionality is nearly free accuracy: part-of-speech and named-entity tagging, framewise phoneme classification, offline speech recognition, protein and DNA sequence labelling, and the encoder of an encoder–decoder model. Bahdanau, Cho and Bengio's attention model uses a bidirectional encoder so that each annotation describes the source word in its full context (see Bahdanau attention). In Transformers the same idea is a mask choice: an encoder lets every token attend to all positions, a decoder masks the future (see Self-Attention).
1import torch, torch.nn as nn23N, T, D, H = 4, 10, 16, 324rnn = nn.RNN(input_size=D, hidden_size=H, num_layers=2,5 bidirectional=True, batch_first=True)6x = torch.randn(N, T, D)7out, h_n = rnn(x) # h_0 defaults to zeros89print(out.shape) # (4, 10, 64) = (N, T, 2*H): [forward ; backward] of the LAST layer10print(h_n.shape) # (4, 4, 32) = (num_layers*2, N, H), index = layer*2 + direction1112hv = h_n.view(2, 2, N, H) # (num_layers, num_directions, N, H)13fwd_last, bwd_last = hv[-1, 0], hv[-1, 1] # same as h_n[-2], h_n[-1]1415# The forward final state sits at the END of the output, the backward one at the START16assert torch.allclose(out[:, -1, :H], fwd_last)17assert torch.allclose(out[:, 0, H:], bwd_last)1819# Sentence vector for classification: both final states, NOT out[:, -1]20sent = torch.cat([fwd_last, bwd_last], dim=-1) # (N, 2H)2122# Parameters: one full set per direction, the reverse ones are suffixed _reverse23print([n for n, _ in rnn.named_parameters()][:8])24# ['weight_ih_l0', 'weight_hh_l0', 'bias_ih_l0', 'bias_hh_l0',25# 'weight_ih_l0_reverse', 'weight_hh_l0_reverse', 'bias_ih_l0_reverse', 'bias_hh_l0_reverse']26print(sum(p.numel() for p in rnn.parameters()))27# 9472 = layer 0: 2*(H*D + H*H + 2H) = 3,200 + layer 1 (reads 2H inputs): 2*(H*2H + H*H + 2H) = 6,272Use when
- +Every output conditions on the full input sequence
- +Two independent chains: simple to train with ordinary BPTT, and parallel on hardware
- +Large, reliable gains on labelling tasks and as a seq2seq encoder
- +Drop-in:
bidirectional=Trueon any RNN, LSTM or GRU layer
Avoid when
- −Needs the whole sequence: no streaming, no autoregressive generation
- −Twice the parameters and compute of one direction; output size doubles to
- −Each half still has the vanishing-gradient reach of its cell
- −Easy to misuse:
out[:, -1], padding and stateful chunking all interact badly with the backward half
Check yourself
You train a character-level language model (predict the next character) with a bidirectional LSTM and its training loss quickly drops almost to zero. What happened?
Check yourself
A 1-layer nn.GRU(input_size=D, hidden_size=H, bidirectional=True, batch_first=True) is applied to a batch of N sequences of length T. What are the shapes of output and h_n?
Remember
- ✓: a forward RNN summarises , an independent backward RNN summarises (Schuster & Paliwal, 1997).
- ✓ for and for ; together they cover the whole input, each with its own decay.
- ✓Needs the full sequence first: fine for tagging, offline ASR and encoders; wrong for language models, decoders and streaming.
- ✓PyTorch: output is , h_n is (layers·2, N, H) with index layer·2 + direction; the sentence vector is , not
out[:, -1]. - ✓Parameters and compute double; interactions between left and right context form in the layer above.