Bidirectional RNN

Two RNNs read the sequence in opposite directions and their states are concatenated, so every output sees the whole input: the past through the forward chain, the future through the backward one. The price is that nothing can be emitted until the sequence has ended.

9 min read991 wordscore0 viewsupdated 2026-10-12

Definition

Bidirectional RNN (BRNN) runs two independent recurrent networks over the same input: a forward RNN from t=1t=1 to TT and a backward RNN from t=Tt=T down to 11. The representation at step tt is the pair yt=[h→t;h←t]y_t=[\overrightarrow h_t;\overleftarrow h_t], which summarises x1,…,xtx_1,\dots,x_t in the first half and xt,…,xTx_t,\dots,x_T in the second.

A unidirectional Recurrent neural network (RNN) compresses only the past into hth_t. That is the right inductive bias for generation, where the future does not exist yet, but it is a handicap for labelling a sequence that is already fully available. To tag the word "bank" as a river bank or a financial bank, or to classify the phoneme at 300 ms of an utterance, the words or frames that come after are as informative as those before.

The obvious fixes were unsatisfying. A time window of ±k\pm k frames fed to a feed-forward net fixes the amount of context in advance. Delaying the output of a forward RNN by dd steps lets it see dd frames of the future, but the network then has to remember the past for dd extra steps and dd must be tuned. Schuster and Paliwal ("Bidirectional recurrent neural networks", IEEE Transactions on Signal Processing, 1997) proposed to split the state into a forward and a backward part with no connections between them, so both can be trained with ordinary backpropagation through time and each output uses all of the input. Baldi, Brunak, Frasconi, Soda and Pollastri (1999) applied the same idea to protein secondary-structure prediction, and Graves and Schmidhuber (2005) combined it with LSTM cells; see Bidirectional LSTM.

Equations

h→t=ϕ(W→xt+U→ h→t−1+b→),h←t=ϕ(W←xt+U← h←t+1+b←),yt=[h→t;h←t]\overrightarrow h_t=\phi\big(\overrightarrow W x_t+\overrightarrow U\,\overrightarrow h_{t-1}+\overrightarrow b\big),\qquad \overleftarrow h_t=\phi\big(\overleftarrow W x_t+\overleftarrow U\,\overleftarrow h_{t+1}+\overleftarrow b\big),\qquad y_t=\big[\overrightarrow h_t;\overleftarrow h_t\big]
h→t\overrightarrow h_t
forward state, a function of x1,…,xtx_1,\dots,x_t; starts from h→0\overrightarrow h_0 (zeros by default)
h←t\overleftarrow h_t
backward state, a function of xt,…,xTx_t,\dots,x_T; starts from h←T+1\overleftarrow h_{T+1}
W→,U→,W←,U←\overrightarrow W,\overrightarrow U,\overleftarrow W,\overleftarrow U
separate weights per direction: nothing is shared between the two chains
ϕ\phi
the cell: tanh⁡\tanh for a vanilla RNN, or an LSTM / GRU cell
yty_t
the concatenation, size 2H2H; a task head reads it, e.g. softmax(Vyt+c)\mathrm{softmax}(V y_t+c) for tagging

Because the two chains never feed into each other, the unrolled graph is still acyclic. Backpropagation through time runs on each chain separately: the gradient of a loss at tt flows left through the forward chain and right through the backward chain. Concatenation is the usual merge; Keras's Bidirectional wrapper also offers sum, mean and product (merge_mode), which keep the size at HH.

At a glance

Context at step t
all of x1..xTx_1..x_T: past via h→t\overrightarrow h_t, future via h←t\overleftarrow h_t
Output size
2H2H per step (concat), HH with sum/mean merge
Parameters
exactly twice a unidirectional layer of the same HH
Compute
two passes of TT steps; they are independent and can run in parallel
Latency
the first output needs the last input: not usable for streaming or for generating text
Typical uses
tagging (POS, NER), speech frame classification, encoders for translation and other seq2seq tasks
h→ₜ = tanh(xₜ + w·h→ₜ₋₁), h←ₜ = tanh(xₜ + w·h←ₜ₊₁), yₜ = [h→ₜ ; h←ₜ], w = 0.90input xₜforward h→backward h←output yₜ0.440.41−0.35−0.92−0.50−0.89−0.09−0.49−0.58−0.85−0.86−0.630.53−0.240.11−0.04−0.25−0.46−0.03−0.25−0.51−0.68−0.72−0.59how much y at t = 4 depends on each input: |∂h→/∂xₛ| uses s ≤ 4, |∂h←/∂xₛ| uses s ≥ 4s=1s=2s=3s=4s=5s=6s=7s=8full sequence: every yₜ sees all 8 inputs (h→ from the left, h← from the right)
click a column to pick yₜ
fig.Two scalar tanh RNNs (input weight 1, recurrent weight w) over 8 random inputs. Click a column: the bars show |∂h→/∂xₛ| (blue, only s ≤ t) and |∂h←/∂xₛ| (orange, only s ≥ t). At the default w = 0.9 the output at t = 4 is y₄ = [−0.86; −0.63], with forward sensitivities 0.09, 0.12, 0.18, 0.26 to x₁…x₄ and backward ones 0.60, 0.53, 0.38, 0.25, 0.15 to x₄…x₈. Lower the tokens-arrived slider to simulate streaming: every backward state is provisional, and when the 7th token has arrived the 7th backward state is still 0.48 away from its final value.

What each half can see

For one layer, the Jacobian of the forward state with respect to an input is a product of one-step Jacobians, ∂h→t/∂xs=(∏k=s+1t∂h→k/∂h→k−1) ∂h→s/∂xs\partial\overrightarrow h_t/\partial x_s=\big(\prod_{k=s+1}^{t}\partial\overrightarrow h_k/\partial\overrightarrow h_{k-1}\big)\,\partial\overrightarrow h_s/\partial x_s for s≤ts\le t, and exactly zero for s>ts>t. The backward state mirrors it. So yty_t is sensitive to every input, but not symmetrically: each half is most sensitive to inputs near tt, and the influence of distant inputs decays with the same vanishing-gradient factors as in any RNN (see Vanilla RNN). A bidirectional net doubles the reach of the context, not its quality.

Notice what yty_t does not contain: no single state has read the whole sequence in one pass. h→t\overrightarrow h_t never saw xt+1x_{t+1} and h←t\overleftarrow h_t never saw xt−1x_{t-1}. Interactions between far-left and far-right context have to be formed by the layer above, which is one reason bidirectional layers are usually stacked (see Stacked RNN): in layer 2 each direction reads the concatenated states of layer 1 and therefore mixes both contexts.

Worked example: three steps by hand

Take scalar states, ϕ=tanh⁡\phi=\tanh, input weight 1, recurrent weight w=0.5w=0.5 in both directions, zero biases and zero initial states, and the sequence x=(1, −0.5, 0.8)x=(1,\,-0.5,\,0.8). Values are rounded to 4 decimals and were checked with the figure's library:

txₜh→ₜ = tanh(xₜ + 0.5·h→ₜ₋₁)h←ₜ = tanh(xₜ + 0.5·h←ₜ₊₁)yₜ
11tanh⁡(1)=0.7616\tanh(1)=0.7616tanh⁡(1−0.0832)=0.7244\tanh(1-0.0832)=0.7244[0.7616; 0.7244]
2−0.5tanh⁡(−0.5+0.3808)=−0.1186\tanh(-0.5+0.3808)=-0.1186tanh⁡(−0.5+0.3320)=−0.1664\tanh(-0.5+0.3320)=-0.1664[−0.1186; −0.1664]
30.8tanh⁡(0.8−0.0593)=0.6296\tanh(0.8-0.0593)=0.6296tanh⁡(0.8)=0.6640\tanh(0.8)=0.6640[0.6296; 0.6640]
Forward, backward and concatenated states (verified in Node)

Sensitivities of y₂, and what streaming would cost

  1. 1Forward half: ∂h→2/∂x2=1−0.11862=0.9859\partial\overrightarrow h_2/\partial x_2=1-0.1186^2=0.9859, ∂h→2/∂x1=0.9859⋅0.5⋅(1−0.76162)=0.2070\partial\overrightarrow h_2/\partial x_1=0.9859\cdot0.5\cdot(1-0.7616^2)=0.2070, and ∂h→2/∂x3=0\partial\overrightarrow h_2/\partial x_3=0.
  2. 2Backward half: ∂h←2/∂x2=1−0.16642=0.9723\partial\overleftarrow h_2/\partial x_2=1-0.1664^2=0.9723, ∂h←2/∂x3=0.9723⋅0.5⋅(1−0.66402)=0.2718\partial\overleftarrow h_2/\partial x_3=0.9723\cdot0.5\cdot(1-0.6640^2)=0.2718, and ∂h←2/∂x1=0\partial\overleftarrow h_2/\partial x_1=0. Finite differences give the same four numbers.
  3. 3Streaming: suppose only x1,x2x_1,x_2 have arrived. The backward chain starts at t=2t=2 and gives h←2=tanh⁡(−0.5)=−0.4621\overleftarrow h_2=\tanh(-0.5)=-0.4621, h←1=0.6463\overleftarrow h_1=0.6463.
  4. 4When x3x_3 arrives, both must be recomputed: h←2\overleftarrow h_2 moves by 0.29570.2957 and h←1\overleftarrow h_1 by 0.07810.0781. Any label emitted early would have been based on wrong states.

When you cannot use it

The backward chain starts at the last input, so a bidirectional layer needs the complete sequence before it can output anything. That rules it out for three common settings:

Where it's used

Language modelling and decoders

Predicting xt+1x_{t+1} from a state that has already read xt+1x_{t+1} is cheating: the backward half would simply copy the answer. Decoders in Sequence-to-sequence (Seq2Seq) models and autoregressive LMs must be causal. (ELMo trains a forward and a backward LM separately for exactly this reason; see Bidirectional LSTM.)

Streaming speech and online control

A live captioner or keyword spotter must emit while audio arrives. Production systems use unidirectional cells, a small fixed look-ahead, or chunked "latency-controlled" BLSTMs (Zhang et al., 2016) whose backward pass only covers a short future chunk.

Long streams with stateful training

The backward state at tt depends on everything up to the end of the stream, so carrying states across chunks (see Truncated backpropagation through time (TBPTT)) only makes sense for the forward direction.

Where the whole input is available, bidirectionality is nearly free accuracy: part-of-speech and named-entity tagging, framewise phoneme classification, offline speech recognition, protein and DNA sequence labelling, and the encoder of an encoder–decoder model. Bahdanau, Cho and Bengio's attention model uses a bidirectional encoder so that each annotation hj=[h→j;h←j]h_j=[\overrightarrow h_j;\overleftarrow h_j] describes the source word jj in its full context (see Bahdanau attention). In Transformers the same idea is a mask choice: an encoder lets every token attend to all positions, a decoder masks the future (see Self-Attention).

python
1import torch, torch.nn as nn
2
3N, T, D, H = 4, 10, 16, 32
4rnn = nn.RNN(input_size=D, hidden_size=H, num_layers=2,
5 bidirectional=True, batch_first=True)
6x = torch.randn(N, T, D)
7out, h_n = rnn(x) # h_0 defaults to zeros
8
9print(out.shape) # (4, 10, 64) = (N, T, 2*H): [forward ; backward] of the LAST layer
10print(h_n.shape) # (4, 4, 32) = (num_layers*2, N, H), index = layer*2 + direction
11
12hv = h_n.view(2, 2, N, H) # (num_layers, num_directions, N, H)
13fwd_last, bwd_last = hv[-1, 0], hv[-1, 1] # same as h_n[-2], h_n[-1]
14
15# The forward final state sits at the END of the output, the backward one at the START
16assert torch.allclose(out[:, -1, :H], fwd_last)
17assert torch.allclose(out[:, 0, H:], bwd_last)
18
19# Sentence vector for classification: both final states, NOT out[:, -1]
20sent = torch.cat([fwd_last, bwd_last], dim=-1) # (N, 2H)
21
22# Parameters: one full set per direction, the reverse ones are suffixed _reverse
23print([n for n, _ in rnn.named_parameters()][:8])
24# ['weight_ih_l0', 'weight_hh_l0', 'bias_ih_l0', 'bias_hh_l0',
25# 'weight_ih_l0_reverse', 'weight_hh_l0_reverse', 'bias_ih_l0_reverse', 'bias_hh_l0_reverse']
26print(sum(p.numel() for p in rnn.parameters()))
27# 9472 = layer 0: 2*(H*D + H*H + 2H) = 3,200 + layer 1 (reads 2H inputs): 2*(H*2H + H*H + 2H) = 6,272
bidirectional=True in PyTorch: output and h_n layout

Use when

  • +Every output conditions on the full input sequence
  • +Two independent chains: simple to train with ordinary BPTT, and parallel on hardware
  • +Large, reliable gains on labelling tasks and as a seq2seq encoder
  • +Drop-in: bidirectional=True on any RNN, LSTM or GRU layer

Avoid when

  • −Needs the whole sequence: no streaming, no autoregressive generation
  • −Twice the parameters and compute of one direction; output size doubles to 2H2H
  • −Each half still has the vanishing-gradient reach of its cell
  • −Easy to misuse: out[:, -1], padding and stateful chunking all interact badly with the backward half

Check yourself

You train a character-level language model (predict the next character) with a bidirectional LSTM and its training loss quickly drops almost to zero. What happened?

Check yourself

A 1-layer nn.GRU(input_size=D, hidden_size=H, bidirectional=True, batch_first=True) is applied to a batch of N sequences of length T. What are the shapes of output and h_n?

Remember

  • ✓yt=[h→t;h←t]y_t=[\overrightarrow h_t;\overleftarrow h_t]: a forward RNN summarises x1..tx_{1..t}, an independent backward RNN summarises xt..Tx_{t..T} (Schuster & Paliwal, 1997).
  • ✓∂h→t/∂xs=0\partial\overrightarrow h_t/\partial x_s=0 for s>ts>t and ∂h←t/∂xs=0\partial\overleftarrow h_t/\partial x_s=0 for s<ts<t; together they cover the whole input, each with its own decay.
  • ✓Needs the full sequence first: fine for tagging, offline ASR and encoders; wrong for language models, decoders and streaming.
  • ✓PyTorch: output is (N,T,2H)(N,T,2H), h_n is (layers·2, N, H) with index layer·2 + direction; the sentence vector is [h→T;h←1][\overrightarrow h_T;\overleftarrow h_1], not out[:, -1].
  • ✓Parameters and compute double; interactions between left and right context form in the layer above.
← back to Deep Learning roadmap