# Tanh

> tanh is a sigmoid rescaled to (−1, 1): zero-centred, with slope 1 at the origin, so it trains better than the sigmoid but still saturates. It remains standard inside RNN and LSTM cells.

Source: https://www.mlfoundry.in/learn/tanh  
Section: Deep Learning › Activation Functions  
Level: core · 9 min read · updated 2026-10-11

**Tanh:** (the **hyperbolic tangent**) is $\tanh(x)=\dfrac{e^{x}-e^{-x}}{e^{x}+e^{-x}}$. It is an S-shaped, odd function with range $(-1,1)$, $\tanh(0)=0$ and slope 1 at the origin, and it equals a rescaled logistic sigmoid: $\tanh(x)=2\sigma(2x)-1$.

Through the 1990s and 2000s tanh was the recommended hidden activation for multilayer perceptrons. LeCun, Bottou, Orr and Müller's "Efficient BackProp" (1998) argued for it over the logistic sigmoid because its outputs are centred on zero. Glorot and Bengio's 2010 initialization analysis was built around it.

ReLU replaced it in deep feedforward and convolutional networks after 2011. Tanh is still standard in recurrent cells: PyTorch's `nn.RNN` uses it by default, and LSTMs and GRUs use it for the candidate state. It also appears wherever an output must lie in $[-1,1]$, such as a value head predicting a game outcome or an image generator's output layer. GPT-2's GELU approximation is written with a tanh.

## Formula and derivative

$$
\tanh(x)=\frac{e^{x}-e^{-x}}{e^{x}+e^{-x}}=2\sigma(2x)-1,\qquad \tanh'(x)=1-\tanh^2(x)=\operatorname{sech}^2(x)\le1
$$

- $2\sigma(2x)-1$: a sigmoid stretched to range $(-1,1)$ and compressed 2× in $x$
- $1-\tanh^2$: computed from the output alone; maximum 1 at x = 0, four times the sigmoid's 0.25
- $\operatorname{sech}$: $1/\cosh$; for large $|x|$, $\tanh'(x)\approx4e^{-2|x|}$

**LeCun's scaled tanh.** "Efficient BackProp" recommends $f(x)=1.7159\tanh\!\big(\tfrac23x\big)$. The constants are chosen so that $f(\pm1)=\pm1$, and the curve is steepest-changing (largest $|f''|$) near $x=\pm1$. Unit-variance inputs then map to outputs of roughly unit variance: $\mathbb{E}[f(u)^2]=0.75$ for $u\sim\mathcal N(0,1)$, versus $0.39$ for plain tanh. Its slope at 0 is $1.7159\cdot\tfrac23=1.144$.

$$
f_{\text{LeCun}}(x)=1.7159\,\tanh\!\Big(\frac{2x}{3}\Big),\qquad f_{\text{LeCun}}(1)=1.0000,\qquad \operatorname{hardtanh}(x)=\max(-1,\min(1,x))
$$

- $1.7159$: the asymptote, chosen so f(1) = 1
- $\operatorname{hardtanh}$: a piecewise-linear approximation with no exp; slope 1 on [−1, 1], 0 outside

**Properties**

- **Range:** $(-1,1)$
- **Monotonic:** strictly increasing
- **Smooth:** yes ($C^\infty$); odd: $\tanh(-x)=-\tanh x$
- **Zero-centred:** yes: $\tanh(0)=0$; $\mathbb{E}[\tanh(u)]=0$ for symmetric inputs
- **Saturating:** both sides: $\tanh'(2)=0.071$, $\tanh'(3)=0.0099$
- **Max slope:** 1 at x = 0 (sigmoid: 0.25)
- **Cost:** exponentials (one exp of 2x, or a library tanh), about the same as a sigmoid

> Figure: tanh and its derivative, with sigmoid and softsign for comparison. tanh's slope peaks at 1, four times the sigmoid's. Softsign has the same slope at 0 but approaches ±1 much more slowly. For Gaussian inputs, tanh's output mean stays at 0 and the sigmoid's sits at 0.5.

## Why zero-centred outputs matter

Consider one unit in the next layer, $y=\sum_i w_ih_i$, receiving the previous layer's activations $h_i$. Its weight gradients are $\partial\mathcal{L}/\partial w_i=\delta\cdot h_i$, where $\delta=\partial\mathcal{L}/\partial y$ is a single number per example.

If every $h_i>0$, as with sigmoid outputs, then all of $\partial\mathcal{L}/\partial w_i$ share the sign of $\delta$. One example can only push all of this unit's weights up together or down together. To raise $w_1$ and lower $w_2$, the optimizer has to alternate between examples, which gives a zig-zag path. LeCun et al. (1998) used this argument to recommend inputs, and activations, with mean close to zero.

The same effect shows up in the loss curvature. The Hessian of a linear unit's squared error is $\mathbb{E}[hh^\top]$. A non-zero mean $\mu$ adds a rank-one term $\mu\mu^\top$, which creates one steep direction along $\mu$ and makes the loss bowl elongated, so gradient descent converges slowly.

> Figure: Two weights learning a teacher w* = (1, −1) from 64 inputs h = φ(u), u ~ N(0, I), starting at (−0.5, 1.5). Left, sigmoid inputs: mean h ≈ (0.52, 0.48), every one-example update (κ = 1) moves both weights in the same direction (60 of 60 steps, inside the shaded quadrants), and the loss needs 36 steps to fall 1000×. Right, tanh inputs: updates point anywhere, and the same drop takes 11 steps. Switch to full-batch updates at the same fraction of each problem's stability limit: the sigmoid's loss bowl has condition number 13.8 and needs 43 steps, while tanh's is nearly round (1.3) and needs 3.

## Effect on training

**Gradient flow.** Near the origin tanh behaves like the identity, so a well-initialized tanh network starts in a nearly linear regime with per-layer gain close to 1, unlike the sigmoid's 0.25. Xavier/Glorot initialization, $\operatorname{Var}(W)=2/(n_{\text{in}}+n_{\text{out}})$, is derived for exactly this regime. PyTorch's `nn.init.calculate_gain('tanh')` returns $5/3$, the extra gain that compensates for tanh's slope falling below 1 away from 0.

**Saturation.** Large pre-activations still kill the gradient: $\tanh'(3)\approx0.01$. Glorot and Bengio (2010) observed that with the older standard initialization, tanh layers saturate one after another during training, starting with the first layer. Normalization layers and careful initialization keep pre-activations in the responsive range.

**Recurrent cells.** In an LSTM, $c_t=f_t\odot c_{t-1}+i_t\odot\tanh(\cdot)$ and $h_t=o_t\odot\tanh(c_t)$. The tanh bounds the candidate and output to $(-1,1)$, so states cannot grow without limit over many time steps. Sigmoid gates decide how much passes. An unbounded activation here would let the recurrence blow up.

## Worked example

tanh and its derivative at a few points (computed in Node; tanh(x) − (2σ(2x) − 1) is below 3e−16 at every point)

| x | −2 | −1 | −0.5 | 0 | 0.5 | 1 | 2 | 3 |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| tanh(x) | −0.9640 | −0.7616 | −0.4621 | 0.0000 | 0.4621 | 0.7616 | 0.9640 | 0.9951 |
| tanh′(x) | 0.0707 | 0.4200 | 0.7864 | 1.0000 | 0.7864 | 0.4200 | 0.0707 | 0.0099 |

A chain of three tanh units with weight $1.5$ each, from $x=0.5$: pre-activations $0.750$, $0.953$, $1.112$, output $0.805$. Backward, the end-to-end derivative is $\prod_k 1.5\,(1-\tanh^2 z_k)=0.320$. The same chain with sigmoids gives $0.0268$, about 12 times smaller, after only three layers.

```python
import torch, torch.nn as nn

x = torch.linspace(-3, 3, 7)

act = nn.Tanh()                       # module
y = torch.tanh(x)                     # functional

def tanh(x):                          # from scratch, via the sigmoid identity (stable for large |x|)
    return 2 * torch.sigmoid(2 * x) - 1

def dtanh(x):
    t = tanh(x)
    return 1 - t * t                  # max 1 at x = 0

def lecun_tanh(x):                    # LeCun et al. 1998: f(±1) = ±1
    return 1.7159 * torch.tanh(2 * x / 3)

# Xavier init paired with tanh
layer = nn.Linear(256, 256)
nn.init.xavier_uniform_(layer.weight, gain=nn.init.calculate_gain("tanh"))   # gain = 5/3

# Recurrent default
rnn = nn.RNN(input_size=32, hidden_size=64, nonlinearity="tanh")
```

tanh in PyTorch: module, functional, from scratch, LeCun's scaled version and Xavier pairing

> **Zero-centred is not non-saturating:** tanh fixes the sigmoid's mean, not its tails. A deep tanh MLP with large initial weights still has most units at $\pm1$ with $\tanh'\approx0$. The fix is Xavier-style initialization (or normalization), not the activation alone.

> **tanh and the sigmoid are the same curve:** $\tanh(x)=2\sigma(2x)-1$, so a tanh network can be rewritten as a sigmoid network with rescaled weights and shifted biases. They differ in the **starting point** gradient descent sees: tanh networks start with zero-mean activations and a slope of 1, which gives a better-conditioned problem.

**Strengths**

- Zero-centred outputs: better-conditioned updates than sigmoid
- Slope 1 at the origin: nearly linear and gain-preserving at good initialization
- Bounded in $(-1,1)$: keeps recurrent states stable
- Smooth, with a derivative computed from the output

**Limitations**

- Saturates for $|x|\gtrsim2$: vanishing gradients in deep stacks
- More expensive than ReLU (exponentials)
- Outperformed by ReLU-family activations in deep feedforward nets and CNNs
- Needs matched initialization (Xavier, gain 5/3)

**Question:** Why do all-positive inputs (like sigmoid outputs) make a unit's weight updates zig-zag?

- A. Positive inputs have higher variance
- B. Each example's gradient δ·hᵢ gives all of that unit's weights the same sign of update, so moving weights in opposite directions takes alternating steps
- C. Positive inputs saturate the next layer
- D. The learning rate must be negative

**Answer:** B. Each example's gradient δ·hᵢ gives all of that unit's weights the same sign of update, so moving weights in opposite directions takes alternating steps — $\partial\mathcal{L}/\partial w_i=\delta\,h_i$ with every $h_i>0$, so the update for one example lies in the all-positive or all-negative orthant. In the figure, all 60 one-example updates with sigmoid inputs had $\Delta w_1\Delta w_2>0$.

**Question:** What is tanh′(0), and how does it compare with σ′(0)?

- A. 0.25, the same as the sigmoid
- B. 1, four times σ′(0) = 0.25
- C. 0.5, twice σ′(0)
- D. 2, eight times σ′(0)

**Answer:** B. 1, four times σ′(0) = 0.25 — $\tanh'(x)=1-\tanh^2x$, which is 1 at $x=0$. From $\tanh(x)=2\sigma(2x)-1$, $\tanh'(x)=4\sigma'(2x)$, so $\tanh'(0)=4\cdot0.25=1$.

**Key points**

- $\tanh(x)=2\sigma(2x)-1\in(-1,1)$, $\tanh'=1-\tanh^2\le1$: four times the sigmoid's peak slope.
- Zero-centred outputs avoid the same-sign update constraint and the ill-conditioning a non-zero mean creates (LeCun et al., 1998).
- LeCun's scaled tanh $1.7159\tanh(2x/3)$ maps $\pm1\to\pm1$; Xavier init (gain $5/3$ in PyTorch) is designed for tanh.
- It still saturates ($\tanh'(3)\approx0.01$), so ReLU-family activations replaced it in deep feedforward nets.
- Still standard inside RNN, LSTM and GRU cells, where its boundedness keeps the recurrence stable, and for outputs in [−1, 1].

---

From MLFoundry (https://www.mlfoundry.in). The web page adds interactive figures and diagrams.
