Tanh

tanh is a sigmoid rescaled to (−1, 1): zero-centred, with slope 1 at the origin, so it trains better than the sigmoid but still saturates. It remains standard inside RNN and LSTM cells.

9 min read674 wordscore0 viewsupdated 2026-10-11

Definition

Tanh (the hyperbolic tangent) is tanh⁡(x)=ex−e−xex+e−x\tanh(x)=\dfrac{e^{x}-e^{-x}}{e^{x}+e^{-x}}. It is an S-shaped, odd function with range (−1,1)(-1,1), tanh⁡(0)=0\tanh(0)=0 and slope 1 at the origin, and it equals a rescaled logistic sigmoid: tanh⁡(x)=2σ(2x)−1\tanh(x)=2\sigma(2x)-1.

Through the 1990s and 2000s tanh was the recommended hidden activation for multilayer perceptrons. LeCun, Bottou, Orr and Müller's "Efficient BackProp" (1998) argued for it over the logistic sigmoid because its outputs are centred on zero. Glorot and Bengio's 2010 initialization analysis was built around it.

ReLU replaced it in deep feedforward and convolutional networks after 2011. Tanh is still standard in recurrent cells: PyTorch's nn.RNN uses it by default, and LSTMs and GRUs use it for the candidate state. It also appears wherever an output must lie in [−1,1][-1,1], such as a value head predicting a game outcome or an image generator's output layer. GPT-2's GELU approximation is written with a tanh.

Formula and derivative

tanh⁡(x)=ex−e−xex+e−x=2σ(2x)−1,tanh⁡′(x)=1−tanh⁡2(x)=sech⁡2(x)≤1\tanh(x)=\frac{e^{x}-e^{-x}}{e^{x}+e^{-x}}=2\sigma(2x)-1,\qquad \tanh'(x)=1-\tanh^2(x)=\operatorname{sech}^2(x)\le1
2σ(2x)−12\sigma(2x)-1
a sigmoid stretched to range (−1,1)(-1,1) and compressed 2× in xx
1−tanh⁡21-\tanh^2
computed from the output alone; maximum 1 at x = 0, four times the sigmoid's 0.25
sech⁡\operatorname{sech}
1/cosh⁡1/\cosh; for large ∣x∣|x|, tanh⁡′(x)≈4e−2∣x∣\tanh'(x)\approx4e^{-2|x|}

LeCun's scaled tanh. "Efficient BackProp" recommends f(x)=1.7159tanh⁡ ⁣(23x)f(x)=1.7159\tanh\!\big(\tfrac23x\big). The constants are chosen so that f(±1)=±1f(\pm1)=\pm1, and the curve is steepest-changing (largest ∣f′′∣|f''|) near x=±1x=\pm1. Unit-variance inputs then map to outputs of roughly unit variance: E[f(u)2]=0.75\mathbb{E}[f(u)^2]=0.75 for u∼N(0,1)u\sim\mathcal N(0,1), versus 0.390.39 for plain tanh. Its slope at 0 is 1.7159⋅23=1.1441.7159\cdot\tfrac23=1.144.

fLeCun(x)=1.7159 tanh⁡ ⁣(2x3),fLeCun(1)=1.0000,hardtanh⁡(x)=max⁡(−1,min⁡(1,x))f_{\text{LeCun}}(x)=1.7159\,\tanh\!\Big(\frac{2x}{3}\Big),\qquad f_{\text{LeCun}}(1)=1.0000,\qquad \operatorname{hardtanh}(x)=\max(-1,\min(1,x))
1.71591.7159
the asymptote, chosen so f(1) = 1
hardtanh⁡\operatorname{hardtanh}
a piecewise-linear approximation with no exp; slope 1 on [−1, 1], 0 outside

Properties

Range
(−1,1)(-1,1)
Monotonic
strictly increasing
Smooth
yes (C∞C^\infty); odd: tanh⁡(−x)=−tanh⁡x\tanh(-x)=-\tanh x
Zero-centred
yes: tanh⁡(0)=0\tanh(0)=0; E[tanh⁡(u)]=0\mathbb{E}[\tanh(u)]=0 for symmetric inputs
Saturating
both sides: tanh⁡′(2)=0.071\tanh'(2)=0.071, tanh⁡′(3)=0.0099\tanh'(3)=0.0099
Max slope
1 at x = 0 (sigmoid: 0.25)
Cost
exponentials (one exp of 2x, or a library tanh), about the same as a sigmoid
f(x) on x ∈ [−5, 5] · drag on the plot to move the probex = 1.00 · f(x) = 0.762 · f′(x) = 0.420−1.0−0.50.00.51.0−4−2024x →TanhTanh f′SigmoidSoftsign
activation
compare
range (−1, 1)monotonic yessmooth yeszero-centred yes
tanh(x) = 2σ(2x) − 1. Maximum slope 1 at x = 0; saturates for large |x|.
signal through the nonlinearity · histogram of f(x) for x ~ N(0.0, 1.00²)exact quadrature, no samplingE[f]−1.000.001.00output value f(x) → (axis spans the central ±4σ of inputs)
E[f(x)] 0.000Var[f(x)] 0.394E[f′(x)] 0.606f′ = 0 for 0.0% of inputsE[f′]¹⁰ 6.6e−3
E[f′]¹⁰ is a rough heuristic for the gradient scale after 10 such layers: it ignores the weights and assumes every layer sees this same input distribution.
fig.tanh and its derivative, with sigmoid and softsign for comparison. tanh's slope peaks at 1, four times the sigmoid's. Softsign has the same slope at 0 but approaches ±1 much more slowly. For Gaussian inputs, tanh's output mean stays at 0 and the sigmoid's sits at 0.5.

Why zero-centred outputs matter

Consider one unit in the next layer, y=∑iwihiy=\sum_i w_ih_i, receiving the previous layer's activations hih_i. Its weight gradients are ∂L/∂wi=δ⋅hi\partial\mathcal{L}/\partial w_i=\delta\cdot h_i, where δ=∂L/∂y\delta=\partial\mathcal{L}/\partial y is a single number per example.

If every hi>0h_i>0, as with sigmoid outputs, then all of ∂L/∂wi\partial\mathcal{L}/\partial w_i share the sign of δ\delta. One example can only push all of this unit's weights up together or down together. To raise w1w_1 and lower w2w_2, the optimizer has to alternate between examples, which gives a zig-zag path. LeCun et al. (1998) used this argument to recommend inputs, and activations, with mean close to zero.

The same effect shows up in the loss curvature. The Hessian of a linear unit's squared error is E[hh⊤]\mathbb{E}[hh^\top]. A non-zero mean μ\mu adds a rank-one term μμ⊤\mu\mu^\top, which creates one steep direction along μ\mu and makes the loss bowl elongated, so gradient descent converges slowly.

inputs h = sigmoid(u): all positivemean h = (0.52, 0.48) · cond = 13.8start✕w* = (1, −1)w₁ →, w₂ ↑loss 0.261 · to 1e−3×: 36 stepssame-sign updates (Δw₁·Δw₂ > 0): 0/0inputs h = tanh(u): zero-centredmean h = (0.04, -0.07) · cond = 1.3start✕w* = (1, −1)w₁ →, w₂ ↑loss 1.57 · to 1e−3×: 11 stepssame-sign updates (Δw₁·Δw₂ > 0): 0/0linear unit y = w·h, teacher t = w*·h, 64 examples · step 0/60 · ellipses: loss contoursshaded: the only directions a one-example update can take when h > 0
updates
playΔw = −κ(y − t)·h/‖h‖²
fig.Two weights learning a teacher w* = (1, −1) from 64 inputs h = φ(u), u ~ N(0, I), starting at (−0.5, 1.5). Left, sigmoid inputs: mean h ≈ (0.52, 0.48), every one-example update (κ = 1) moves both weights in the same direction (60 of 60 steps, inside the shaded quadrants), and the loss needs 36 steps to fall 1000×. Right, tanh inputs: updates point anywhere, and the same drop takes 11 steps. Switch to full-batch updates at the same fraction of each problem's stability limit: the sigmoid's loss bowl has condition number 13.8 and needs 43 steps, while tanh's is nearly round (1.3) and needs 3.

Effect on training

Gradient flow. Near the origin tanh behaves like the identity, so a well-initialized tanh network starts in a nearly linear regime with per-layer gain close to 1, unlike the sigmoid's 0.25. Xavier/Glorot initialization, Var⁡(W)=2/(nin+nout)\operatorname{Var}(W)=2/(n_{\text{in}}+n_{\text{out}}), is derived for exactly this regime. PyTorch's nn.init.calculate_gain('tanh') returns 5/35/3, the extra gain that compensates for tanh's slope falling below 1 away from 0.

Saturation. Large pre-activations still kill the gradient: tanh⁡′(3)≈0.01\tanh'(3)\approx0.01. Glorot and Bengio (2010) observed that with the older standard initialization, tanh layers saturate one after another during training, starting with the first layer. Normalization layers and careful initialization keep pre-activations in the responsive range.

Recurrent cells. In an LSTM, ct=ft⊙ct−1+it⊙tanh⁡(⋅)c_t=f_t\odot c_{t-1}+i_t\odot\tanh(\cdot) and ht=ot⊙tanh⁡(ct)h_t=o_t\odot\tanh(c_t). The tanh bounds the candidate and output to (−1,1)(-1,1), so states cannot grow without limit over many time steps. Sigmoid gates decide how much passes. An unbounded activation here would let the recurrence blow up.

Worked example

x−2−1−0.500.5123
tanh(x)−0.9640−0.7616−0.46210.00000.46210.76160.96400.9951
tanh′(x)0.07070.42000.78641.00000.78640.42000.07070.0099
tanh and its derivative at a few points (computed in Node; tanh(x) − (2σ(2x) − 1) is below 3e−16 at every point)

A chain of three tanh units with weight 1.51.5 each, from x=0.5x=0.5: pre-activations 0.7500.750, 0.9530.953, 1.1121.112, output 0.8050.805. Backward, the end-to-end derivative is ∏k1.5 (1−tanh⁡2zk)=0.320\prod_k 1.5\,(1-\tanh^2 z_k)=0.320. The same chain with sigmoids gives 0.02680.0268, about 12 times smaller, after only three layers.

python
1import torch, torch.nn as nn
2
3x = torch.linspace(-3, 3, 7)
4
5act = nn.Tanh() # module
6y = torch.tanh(x) # functional
7
8def tanh(x): # from scratch, via the sigmoid identity (stable for large |x|)
9 return 2 * torch.sigmoid(2 * x) - 1
10
11def dtanh(x):
12 t = tanh(x)
13 return 1 - t * t # max 1 at x = 0
14
15def lecun_tanh(x): # LeCun et al. 1998: f(±1) = ±1
16 return 1.7159 * torch.tanh(2 * x / 3)
17
18# Xavier init paired with tanh
19layer = nn.Linear(256, 256)
20nn.init.xavier_uniform_(layer.weight, gain=nn.init.calculate_gain("tanh")) # gain = 5/3
21
22# Recurrent default
23rnn = nn.RNN(input_size=32, hidden_size=64, nonlinearity="tanh")
tanh in PyTorch: module, functional, from scratch, LeCun's scaled version and Xavier pairing

Use when

  • +Zero-centred outputs: better-conditioned updates than sigmoid
  • +Slope 1 at the origin: nearly linear and gain-preserving at good initialization
  • +Bounded in (−1,1)(-1,1): keeps recurrent states stable
  • +Smooth, with a derivative computed from the output

Avoid when

  • −Saturates for ∣x∣≳2|x|\gtrsim2: vanishing gradients in deep stacks
  • −More expensive than ReLU (exponentials)
  • −Outperformed by ReLU-family activations in deep feedforward nets and CNNs
  • −Needs matched initialization (Xavier, gain 5/3)

Check yourself

Why do all-positive inputs (like sigmoid outputs) make a unit's weight updates zig-zag?

Check yourself

What is tanh′(0), and how does it compare with σ′(0)?

Remember

  • ✓tanh⁡(x)=2σ(2x)−1∈(−1,1)\tanh(x)=2\sigma(2x)-1\in(-1,1), tanh⁡′=1−tanh⁡2≤1\tanh'=1-\tanh^2\le1: four times the sigmoid's peak slope.
  • ✓Zero-centred outputs avoid the same-sign update constraint and the ill-conditioning a non-zero mean creates (LeCun et al., 1998).
  • ✓LeCun's scaled tanh 1.7159tanh⁡(2x/3)1.7159\tanh(2x/3) maps ±1→±1\pm1\to\pm1; Xavier init (gain 5/35/3 in PyTorch) is designed for tanh.
  • ✓It still saturates (tanh⁡′(3)≈0.01\tanh'(3)\approx0.01), so ReLU-family activations replaced it in deep feedforward nets.
  • ✓Still standard inside RNN, LSTM and GRU cells, where its boundedness keeps the recurrence stable, and for outputs in [−1, 1].
← back to Deep Learning roadmap