# Bidirectional RNN

> Two RNNs read the sequence in opposite directions and their states are concatenated, so every output sees the whole input: the past through the forward chain, the future through the backward one. The price is that nothing can be emitted until the sequence has ended.

Source: https://www.mlfoundry.in/learn/bidirectional-rnn  
Section: Deep Learning › Recurrent Neural Networks and Sequence Models  
Level: core · 9 min read · updated 2026-10-12

**Bidirectional RNN:** (BRNN) runs two independent recurrent networks over the same input: a **forward** RNN from $t=1$ to $T$ and a **backward** RNN from $t=T$ down to $1$. The representation at step $t$ is the pair $y_t=[\overrightarrow h_t;\overleftarrow h_t]$, which summarises $x_1,\dots,x_t$ in the first half and $x_t,\dots,x_T$ in the second.

A unidirectional **Recurrent neural network (RNN)** compresses only the past into $h_t$. That is the right inductive bias for generation, where the future does not exist yet, but it is a handicap for **labelling** a sequence that is already fully available. To tag the word "bank" as a river bank or a financial bank, or to classify the phoneme at 300 ms of an utterance, the words or frames that come **after** are as informative as those before.

The obvious fixes were unsatisfying. A time window of $\pm k$ frames fed to a feed-forward net fixes the amount of context in advance. Delaying the output of a forward RNN by $d$ steps lets it see $d$ frames of the future, but the network then has to remember the past for $d$ extra steps and $d$ must be tuned. Schuster and Paliwal ("Bidirectional recurrent neural networks", *IEEE Transactions on Signal Processing*, 1997) proposed to split the state into a forward and a backward part with no connections between them, so both can be trained with ordinary backpropagation through time and each output uses all of the input. Baldi, Brunak, Frasconi, Soda and Pollastri (1999) applied the same idea to protein secondary-structure prediction, and Graves and Schmidhuber (2005) combined it with LSTM cells; see **Bidirectional LSTM**.

## Equations

$$
\overrightarrow h_t=\phi\big(\overrightarrow W x_t+\overrightarrow U\,\overrightarrow h_{t-1}+\overrightarrow b\big),\qquad \overleftarrow h_t=\phi\big(\overleftarrow W x_t+\overleftarrow U\,\overleftarrow h_{t+1}+\overleftarrow b\big),\qquad y_t=\big[\overrightarrow h_t;\overleftarrow h_t\big]
$$

- $\overrightarrow h_t$: forward state, a function of $x_1,\dots,x_t$; starts from $\overrightarrow h_0$ (zeros by default)
- $\overleftarrow h_t$: backward state, a function of $x_t,\dots,x_T$; starts from $\overleftarrow h_{T+1}$
- $\overrightarrow W,\overrightarrow U,\overleftarrow W,\overleftarrow U$: separate weights per direction: nothing is shared between the two chains
- $\phi$: the cell: $\tanh$ for a vanilla RNN, or an LSTM / GRU cell
- $y_t$: the concatenation, size $2H$; a task head reads it, e.g. $\mathrm{softmax}(V y_t+c)$ for tagging

Because the two chains never feed into each other, the unrolled graph is still acyclic. Backpropagation through time runs on each chain separately: the gradient of a loss at $t$ flows left through the forward chain and right through the backward chain. Concatenation is the usual merge; Keras's `Bidirectional` wrapper also offers sum, mean and product (`merge_mode`), which keep the size at $H$.

**At a glance**

- **Context at step t:** all of $x_1..x_T$: past via $\overrightarrow h_t$, future via $\overleftarrow h_t$
- **Output size:** $2H$ per step (concat), $H$ with sum/mean merge
- **Parameters:** exactly twice a unidirectional layer of the same $H$
- **Compute:** two passes of $T$ steps; they are independent and can run in parallel
- **Latency:** the first output needs the last input: not usable for streaming or for generating text
- **Typical uses:** tagging (POS, NER), speech frame classification, encoders for translation and other seq2seq tasks

> Figure: Two scalar tanh RNNs (input weight 1, recurrent weight w) over 8 random inputs. Click a column: the bars show |∂h→/∂xₛ| (blue, only s ≤ t) and |∂h←/∂xₛ| (orange, only s ≥ t). At the default w = 0.9 the output at t = 4 is y₄ = [−0.86; −0.63], with forward sensitivities 0.09, 0.12, 0.18, 0.26 to x₁…x₄ and backward ones 0.60, 0.53, 0.38, 0.25, 0.15 to x₄…x₈. Lower the tokens-arrived slider to simulate streaming: every backward state is provisional, and when the 7th token has arrived the 7th backward state is still 0.48 away from its final value.

## What each half can see

For one layer, the Jacobian of the forward state with respect to an input is a product of one-step Jacobians, $\partial\overrightarrow h_t/\partial x_s=\big(\prod_{k=s+1}^{t}\partial\overrightarrow h_k/\partial\overrightarrow h_{k-1}\big)\,\partial\overrightarrow h_s/\partial x_s$ for $s\le t$, and exactly zero for $s>t$. The backward state mirrors it. So $y_t$ is sensitive to every input, but **not symmetrically**: each half is most sensitive to inputs near $t$, and the influence of distant inputs decays with the same vanishing-gradient factors as in any RNN (see **Vanilla RNN**). A bidirectional net doubles the reach of the context, not its quality.

Notice what $y_t$ does **not** contain: no single state has read the whole sequence in one pass. $\overrightarrow h_t$ never saw $x_{t+1}$ and $\overleftarrow h_t$ never saw $x_{t-1}$. Interactions between far-left and far-right context have to be formed by the layer **above**, which is one reason bidirectional layers are usually stacked (see **Stacked RNN**): in layer 2 each direction reads the concatenated states of layer 1 and therefore mixes both contexts.

## Worked example: three steps by hand

Take scalar states, $\phi=\tanh$, input weight 1, recurrent weight $w=0.5$ in both directions, zero biases and zero initial states, and the sequence $x=(1,\,-0.5,\,0.8)$. Values are rounded to 4 decimals and were checked with the figure's library:

Forward, backward and concatenated states (verified in Node)

| t | xₜ | h→ₜ = tanh(xₜ + 0.5·h→ₜ₋₁) | h←ₜ = tanh(xₜ + 0.5·h←ₜ₊₁) | yₜ |
| --- | --- | --- | --- | --- |
| 1 | 1 | $\tanh(1)=0.7616$ | $\tanh(1-0.0832)=0.7244$ | [0.7616; 0.7244] |
| 2 | −0.5 | $\tanh(-0.5+0.3808)=-0.1186$ | $\tanh(-0.5+0.3320)=-0.1664$ | [−0.1186; −0.1664] |
| 3 | 0.8 | $\tanh(0.8-0.0593)=0.6296$ | $\tanh(0.8)=0.6640$ | [0.6296; 0.6640] |

**Sensitivities of y₂, and what streaming would cost**

1. Forward half: $\partial\overrightarrow h_2/\partial x_2=1-0.1186^2=0.9859$, $\partial\overrightarrow h_2/\partial x_1=0.9859\cdot0.5\cdot(1-0.7616^2)=0.2070$, and $\partial\overrightarrow h_2/\partial x_3=0$.
2. Backward half: $\partial\overleftarrow h_2/\partial x_2=1-0.1664^2=0.9723$, $\partial\overleftarrow h_2/\partial x_3=0.9723\cdot0.5\cdot(1-0.6640^2)=0.2718$, and $\partial\overleftarrow h_2/\partial x_1=0$. Finite differences give the same four numbers.
3. Streaming: suppose only $x_1,x_2$ have arrived. The backward chain starts at $t=2$ and gives $\overleftarrow h_2=\tanh(-0.5)=-0.4621$, $\overleftarrow h_1=0.6463$.
4. When $x_3$ arrives, both must be recomputed: $\overleftarrow h_2$ moves by $0.2957$ and $\overleftarrow h_1$ by $0.0781$. Any label emitted early would have been based on wrong states.

## When you cannot use it

The backward chain starts at the **last** input, so a bidirectional layer needs the complete sequence before it can output anything. That rules it out for three common settings:

- **Language modelling and decoders:** Predicting $x_{t+1}$ from a state that has already read $x_{t+1}$ is cheating: the backward half would simply copy the answer. Decoders in **Sequence-to-sequence (Seq2Seq)** models and autoregressive LMs must be causal. (ELMo trains a forward and a backward LM *separately* for exactly this reason; see **Bidirectional LSTM**.)
- **Streaming speech and online control:** A live captioner or keyword spotter must emit while audio arrives. Production systems use unidirectional cells, a small fixed look-ahead, or chunked "latency-controlled" BLSTMs (Zhang et al., 2016) whose backward pass only covers a short future chunk.
- **Long streams with stateful training:** The backward state at $t$ depends on everything up to the end of the stream, so carrying states across chunks (see **Truncated backpropagation through time (TBPTT)**) only makes sense for the forward direction.

Where the whole input *is* available, bidirectionality is nearly free accuracy: part-of-speech and named-entity tagging, framewise phoneme classification, offline speech recognition, protein and DNA sequence labelling, and the **encoder** of an encoder–decoder model. Bahdanau, Cho and Bengio's attention model uses a bidirectional encoder so that each annotation $h_j=[\overrightarrow h_j;\overleftarrow h_j]$ describes the source word $j$ in its full context (see **Bahdanau attention**). In Transformers the same idea is a mask choice: an encoder lets every token attend to all positions, a decoder masks the future (see **Self-Attention**).

```python
import torch, torch.nn as nn

N, T, D, H = 4, 10, 16, 32
rnn = nn.RNN(input_size=D, hidden_size=H, num_layers=2,
             bidirectional=True, batch_first=True)
x = torch.randn(N, T, D)
out, h_n = rnn(x)                    # h_0 defaults to zeros

print(out.shape)   # (4, 10, 64)  = (N, T, 2*H): [forward ; backward] of the LAST layer
print(h_n.shape)   # (4, 4, 32)   = (num_layers*2, N, H), index = layer*2 + direction

hv = h_n.view(2, 2, N, H)            # (num_layers, num_directions, N, H)
fwd_last, bwd_last = hv[-1, 0], hv[-1, 1]         # same as h_n[-2], h_n[-1]

# The forward final state sits at the END of the output, the backward one at the START
assert torch.allclose(out[:, -1, :H], fwd_last)
assert torch.allclose(out[:, 0, H:], bwd_last)

# Sentence vector for classification: both final states, NOT out[:, -1]
sent = torch.cat([fwd_last, bwd_last], dim=-1)    # (N, 2H)

# Parameters: one full set per direction, the reverse ones are suffixed _reverse
print([n for n, _ in rnn.named_parameters()][:8])
# ['weight_ih_l0', 'weight_hh_l0', 'bias_ih_l0', 'bias_hh_l0',
#  'weight_ih_l0_reverse', 'weight_hh_l0_reverse', 'bias_ih_l0_reverse', 'bias_hh_l0_reverse']
print(sum(p.numel() for p in rnn.parameters()))
# 9472 = layer 0: 2*(H*D + H*H + 2H) = 3,200  +  layer 1 (reads 2H inputs): 2*(H*2H + H*H + 2H) = 6,272
```

bidirectional=True in PyTorch: output and h_n layout

> **out[:, -1] is not a sentence vector:** For a bidirectional layer, `out[:, -1]` concatenates the forward state after the **whole** sequence with the backward state after **one** token. Use $[\overrightarrow h_T;\overleftarrow h_1]$ from `h_n` instead, or pool over time. With padded batches even `out[:, -1, :H]` is wrong for short sequences; pack them first (see **Bidirectional LSTM**).

> **Two RNNs, not one RNN with more context:** A BRNN is mathematically just two unidirectional RNNs whose outputs are concatenated, so every result about one chain (vanishing gradients, gating, clipping) applies to each half. What changes is the *task contract*: the model may condition on the future. That is why it helps tagging a lot and helps language modelling not at all.

> **Look-ahead as a middle ground:** If you need low latency but some future context helps, a unidirectional RNN whose output for step $t$ is read at step $t+d$ gives $d$ frames of look-ahead at a latency of $d$ frames. Schuster and Paliwal compared exactly this delayed forward RNN with the bidirectional one; the BRNN avoids choosing $d$ and does not force the forward chain to carry the past for $d$ extra steps.

**Strengths**

- Every output conditions on the full input sequence
- Two independent chains: simple to train with ordinary BPTT, and parallel on hardware
- Large, reliable gains on labelling tasks and as a seq2seq encoder
- Drop-in: `bidirectional=True` on any RNN, LSTM or GRU layer

**Limitations**

- Needs the whole sequence: no streaming, no autoregressive generation
- Twice the parameters and compute of one direction; output size doubles to $2H$
- Each half still has the vanishing-gradient reach of its cell
- Easy to misuse: `out[:, -1]`, padding and stateful chunking all interact badly with the backward half

**Question:** You train a character-level language model (predict the next character) with a bidirectional LSTM and its training loss quickly drops almost to zero. What happened?

- A. The backward half has already read the character it is asked to predict, so it leaks the answer
- B. Bidirectional models are simply stronger language models
- C. The forward and backward weights became tied
- D. The learning rate was too high

**Answer:** A. The backward half has already read the character it is asked to predict, so it leaks the answer — At step $t$ the backward state $\overleftarrow h_t$ is computed from $x_t,\dots,x_T$, which includes the target $x_{t+1}$. The model learns to copy it. Language models must be causal; ELMo uses a forward LM and a separate backward LM, each predicting only from its own side.

**Question:** A 1-layer nn.GRU(input_size=D, hidden_size=H, bidirectional=True, batch_first=True) is applied to a batch of N sequences of length T. What are the shapes of output and h_n?

- A. (N, T, 2H) and (2, N, H)
- B. (N, T, H) and (2, N, H)
- C. (N, T, 2H) and (N, 2H)
- D. (T, N, 2H) and (1, N, 2H)

**Answer:** A. (N, T, 2H) and (2, N, H) — `output` concatenates both directions at each step, so its last dimension is $2H$; `batch_first=True` puts $N$ first. `h_n` is never batch-first: it is (num_layers·num_directions, N, H), here (2, N, H) with index 0 = forward, 1 = backward.

**Key points**

- $y_t=[\overrightarrow h_t;\overleftarrow h_t]$: a forward RNN summarises $x_{1..t}$, an independent backward RNN summarises $x_{t..T}$ (Schuster & Paliwal, 1997).
- $\partial\overrightarrow h_t/\partial x_s=0$ for $s>t$ and $\partial\overleftarrow h_t/\partial x_s=0$ for $s<t$; together they cover the whole input, each with its own decay.
- Needs the full sequence first: fine for tagging, offline ASR and encoders; wrong for language models, decoders and streaming.
- PyTorch: output is $(N,T,2H)$, h_n is (layers·2, N, H) with index layer·2 + direction; the sentence vector is $[\overrightarrow h_T;\overleftarrow h_1]$, not `out[:, -1]`.
- Parameters and compute double; interactions between left and right context form in the layer above.

---

From MLFoundry (https://www.mlfoundry.in). The web page adds interactive figures and diagrams.
