tanh is a sigmoid rescaled to (−1, 1): zero-centred, with slope 1 at the origin, so it trains better than the sigmoid but still saturates. It remains standard inside RNN and LSTM cells.
9 min read674 wordscore0 viewsupdated 2026-10-11
Definition
Tanh (the hyperbolic tangent) is tanh(x)=ex+e−xex−e−x. It is an S-shaped, odd function with range (−1,1), tanh(0)=0 and slope 1 at the origin, and it equals a rescaled logistic sigmoid: tanh(x)=2σ(2x)−1.
Through the 1990s and 2000s tanh was the recommended hidden activation for multilayer perceptrons. LeCun, Bottou, Orr and Müller's "Efficient BackProp" (1998) argued for it over the logistic sigmoid because its outputs are centred on zero. Glorot and Bengio's 2010 initialization analysis was built around it.
ReLU replaced it in deep feedforward and convolutional networks after 2011. Tanh is still standard in recurrent cells: PyTorch's nn.RNN uses it by default, and LSTMs and GRUs use it for the candidate state. It also appears wherever an output must lie in [−1,1], such as a value head predicting a game outcome or an image generator's output layer. GPT-2's GELU approximation is written with a tanh.
a sigmoid stretched to range (−1,1) and compressed 2× in x
1−tanh2
computed from the output alone; maximum 1 at x = 0, four times the sigmoid's 0.25
sech
1/cosh; for large ∣x∣, tanh′(x)≈4e−2∣x∣
LeCun's scaled tanh. "Efficient BackProp" recommends f(x)=1.7159tanh(32x). The constants are chosen so that f(±1)=±1, and the curve is steepest-changing (largest ∣f′′∣) near x=±1. Unit-variance inputs then map to outputs of roughly unit variance: E[f(u)2]=0.75 for u∼N(0,1), versus 0.39 for plain tanh. Its slope at 0 is 1.7159⋅32=1.144.
a piecewise-linear approximation with no exp; slope 1 on [−1, 1], 0 outside
Properties
Range
(−1,1)
Monotonic
strictly increasing
Smooth
yes (C∞); odd: tanh(−x)=−tanhx
Zero-centred
yes: tanh(0)=0; E[tanh(u)]=0 for symmetric inputs
Saturating
both sides: tanh′(2)=0.071, tanh′(3)=0.0099
Max slope
1 at x = 0 (sigmoid: 0.25)
Cost
exponentials (one exp of 2x, or a library tanh), about the same as a sigmoid
activation
compare
range (−1, 1)monotonic yessmooth yeszero-centred yes
tanh(x) = 2σ(2x) − 1. Maximum slope 1 at x = 0; saturates for large |x|.
E[f(x)] 0.000Var[f(x)] 0.394E[f′(x)] 0.606f′ = 0 for 0.0% of inputsE[f′]¹⁰ 6.6e−3
E[f′]¹⁰ is a rough heuristic for the gradient scale after 10 such layers: it ignores the weights and assumes every layer sees this same input distribution.
fig.tanh and its derivative, with sigmoid and softsign for comparison. tanh's slope peaks at 1, four times the sigmoid's. Softsign has the same slope at 0 but approaches ±1 much more slowly. For Gaussian inputs, tanh's output mean stays at 0 and the sigmoid's sits at 0.5.
Why zero-centred outputs matter
Consider one unit in the next layer, y=∑iwihi, receiving the previous layer's activations hi. Its weight gradients are ∂L/∂wi=δ⋅hi, where δ=∂L/∂y is a single number per example.
If every hi>0, as with sigmoid outputs, then all of ∂L/∂wi share the sign of δ. One example can only push all of this unit's weights up together or down together. To raise w1 and lower w2, the optimizer has to alternate between examples, which gives a zig-zag path. LeCun et al. (1998) used this argument to recommend inputs, and activations, with mean close to zero.
The same effect shows up in the loss curvature. The Hessian of a linear unit's squared error is E[hh⊤]. A non-zero mean μ adds a rank-one term μμ⊤, which creates one steep direction along μ and makes the loss bowl elongated, so gradient descent converges slowly.
updates
playΔw = −κ(y − t)·h/‖h‖²
fig.Two weights learning a teacher w* = (1, −1) from 64 inputs h = φ(u), u ~ N(0, I), starting at (−0.5, 1.5). Left, sigmoid inputs: mean h ≈ (0.52, 0.48), every one-example update (κ = 1) moves both weights in the same direction (60 of 60 steps, inside the shaded quadrants), and the loss needs 36 steps to fall 1000×. Right, tanh inputs: updates point anywhere, and the same drop takes 11 steps. Switch to full-batch updates at the same fraction of each problem's stability limit: the sigmoid's loss bowl has condition number 13.8 and needs 43 steps, while tanh's is nearly round (1.3) and needs 3.
Effect on training
Gradient flow. Near the origin tanh behaves like the identity, so a well-initialized tanh network starts in a nearly linear regime with per-layer gain close to 1, unlike the sigmoid's 0.25. Xavier/Glorot initialization, Var(W)=2/(nin+nout), is derived for exactly this regime. PyTorch's nn.init.calculate_gain('tanh') returns 5/3, the extra gain that compensates for tanh's slope falling below 1 away from 0.
Saturation. Large pre-activations still kill the gradient: tanh′(3)≈0.01. Glorot and Bengio (2010) observed that with the older standard initialization, tanh layers saturate one after another during training, starting with the first layer. Normalization layers and careful initialization keep pre-activations in the responsive range.
Recurrent cells. In an LSTM, ct=ft⊙ct−1+it⊙tanh(⋅) and ht=ot⊙tanh(ct). The tanh bounds the candidate and output to (−1,1), so states cannot grow without limit over many time steps. Sigmoid gates decide how much passes. An unbounded activation here would let the recurrence blow up.
Worked example
x
−2
−1
−0.5
0
0.5
1
2
3
tanh(x)
−0.9640
−0.7616
−0.4621
0.0000
0.4621
0.7616
0.9640
0.9951
tanh′(x)
0.0707
0.4200
0.7864
1.0000
0.7864
0.4200
0.0707
0.0099
tanh and its derivative at a few points (computed in Node; tanh(x) − (2σ(2x) − 1) is below 3e−16 at every point)
A chain of three tanh units with weight 1.5 each, from x=0.5: pre-activations 0.750, 0.953, 1.112, output 0.805. Backward, the end-to-end derivative is ∏k1.5(1−tanh2zk)=0.320. The same chain with sigmoids gives 0.0268, about 12 times smaller, after only three layers.
python
1import torch, torch.nn as nn
2
3x = torch.linspace(-3, 3, 7)
4
5act = nn.Tanh() # module
6y = torch.tanh(x) # functional
7
8deftanh(x): # from scratch, via the sigmoid identity (stable for large |x|)
9return2 * torch.sigmoid(2 * x) - 1
10
11defdtanh(x):
12 t = tanh(x)
13return1 - t * t # max 1 at x = 0
14
15deflecun_tanh(x): # LeCun et al. 1998: f(±1) = ±1
16return1.7159 * torch.tanh(2 * x / 3)
17
18# Xavier init paired with tanh
19layer = nn.Linear(256, 256)
20nn.init.xavier_uniform_(layer.weight, gain=nn.init.calculate_gain("tanh")) # gain = 5/3