Linear activation

f(x) = ax passes the input through unchanged. Stacked linear layers collapse into one matrix, so it adds no expressive power, but it is right for outputs, projections and bottlenecks.

8 min read539 wordscore0 viewsupdated 2026-10-11

Definition

Linear activation (also identity activation, or "no activation") is f(x)=axf(x)=ax, usually with a=1a=1. A layer with a linear activation computes an affine map Wx+bW x+b and nothing more.

Every other activation in this section exists because of one fact about this one: composing linear maps gives a linear map. A network with only linear activations, however deep and wide, computes exactly what a single linear layer computes. Depth adds parameters but no expressive power.

That does not make the linear activation useless. Wherever a layer should not bend its input, it is the correct choice: the output of a regression model, the logits fed to a softmax, the Q/K/V projections in attention, and the narrow bottleneck layers of MobileNetV2. "Linear activation" mostly means leaving the nonlinearity out on purpose.

Formula and derivative

f(x)=a x,f′(x)=af(x)=a\,x,\qquad f'(x)=a
aa
slope; almost always 1 (the identity). Any other value can be absorbed into the next layer's weights
f′f'
constant: every unit passes the gradient back scaled by exactly a, at every input

Properties

Range
(−∞,∞)(-\infty,\infty), unbounded
Monotonic
yes for a>0a>0
Smooth
yes (C∞C^\infty); f′′=0f''=0
Zero-centred
yes: odd function, f(0) = 0
Saturating
never, so it neither vanishes nor bounds activations
Cost
free: no operation at all for a = 1
Expressive power
none added: any stack of linear layers equals one linear layer
f(x) on x ∈ [−5, 5] · drag on the plot to move the probex = 1.00 · f(x) = 1.000 · f′(x) = 1.000−4−2024−4−2024x →Linear (a=1)Linear f′Tanh
activation
compare
range (−∞, ∞)monotonic yessmooth yeszero-centred yes
Identity when a = 1. Monotonic only for a ≥ 0. Stacking linear layers stays linear.
signal through the nonlinearity · histogram of f(x) for x ~ N(0.0, 1.00²)exact quadrature, no samplingE[f]−4.000.004.00output value f(x) → (axis spans the central ±4σ of inputs)
E[f(x)] 0.000Var[f(x)] 1.000E[f′(x)] 1.000f′ = 0 for 0.0% of inputsE[f′]¹⁰ 1.000
E[f′]¹⁰ is a rough heuristic for the gradient scale after 10 such layers: it ignores the weights and assumes every layer sees this same input distribution.
fig.The identity and its constant derivative, with tanh for contrast: tanh agrees with the identity near 0 and bends away from it as |x| grows. For Gaussian inputs, the identity's output statistics are just the input's.

The collapse of stacked linear layers

Two layers with linear activations, with the slope aa absorbed into the weights:

W2(W1x+b1)+b2=(W2W1)⏟A x+(W2b1+b2)⏟cW_2\big(W_1x+b_1\big)+b_2=\underbrace{(W_2W_1)}_{A}\,x+\underbrace{(W_2b_1+b_2)}_{c}
AA
the product matrix; for LL layers A=aL WL⋯W1A=a^L\,W_L\cdots W_1
cc
all the biases, pushed through the later layers

By induction, any number of layers collapses to one affine map Ax+cAx+c. Its decision boundaries are hyperplanes, the same ones logistic regression can draw. Its rank is at most the width of the narrowest layer: a 1024→64→10241024\to64\to1024 linear stack is a rank-64 matrix. That is a constraint, not extra capacity.

input xafter layer 3.00 of 3 · φ = 1.00·x3×→φ(Wh+b)view (−1.06, 0.05) ± 1.50╌ A x + c with A = 1.00^3·W_3⋯W_1 = [−0.46 0.18; −0.36 −0.55], c = (−1.06, 0.05)det A = 0.318 · max ‖stack(x) − (A x + c)‖ over the grid = 6.0e-16linear: lines stay straight and parallel, the stack equals the dashed map
activation
fig.A grid and two rings of points pushed through L random 2-D layers h ← φ(Wh + b), animated layer by layer. With φ(x) = a·x, the output grid stays straight and evenly spaced, and it lands exactly on the dashed grid of one affine map Ax + c: at the default seed with L = 3 the largest gap is 6.0e−16, i.e. floating-point rounding. No line can separate the inner ring from the outer one before or after. Switch to tanh: the lines bend, and the same product of matrices misses the true output by up to 1.40.

Worked example

Take W1=[21−11]W_1=\begin{bmatrix}2&1\\-1&1\end{bmatrix}, b1=(1,0)b_1=(1,0), W2=[1−102]W_2=\begin{bmatrix}1&-1\\0&2\end{bmatrix}, b2=(0,1)b_2=(0,1) and x=(1,2)x=(1,2):

QuantityComputationValue
h1h_1W1x+b1=(2+2+1,  −1+2)W_1x+b_1=(2+2+1,\;-1+2)(5,  1)(5,\;1)
yyW2h1+b2=(5−1,  2+1)W_2h_1+b_2=(5-1,\;2+1)(4,  3)(4,\;3)
A=W2W1A=W_2W_1[1⋅2+(−1)(−1)1−10⋅2+2(−1)0+2]\begin{bmatrix}1\cdot2+(-1)(-1) & 1-1\\ 0\cdot2+2(-1) & 0+2\end{bmatrix}[30−22]\begin{bmatrix}3&0\\-2&2\end{bmatrix}
c=W2b1+b2c=W_2b_1+b_2(1,  0)+(0,  1)(1,\;0)+(0,\;1)(1,  1)(1,\;1)
Ax+cAx+c(3+1,  −2+4+1)(3+1,\;-2+4+1)(4,  3)(4,\;3) — identical
with tanh after layer 1W2tanh⁡(h1)+b2W_2\tanh(h_1)+b_2, tanh⁡(h1)=(0.9999, 0.7616)\tanh(h_1)=(0.9999,\,0.7616)(0.2383,  2.5232)(0.2383,\;2.5232)
Layer by layer versus the collapsed map, and the same weights with tanh in between (computed in Node)

The second layer added parameters but nothing a single 2×22\times2 matrix could not do. With tanh in between, the output changes completely, and no single matrix maps every xx to it.

Effect on training

Gradients through a linear unit are simple: f′=af'=a everywhere, so there is no saturation and no dead region. A deep linear network still multiplies one weight matrix per layer in the backward pass, so its gradients can vanish or explode exactly as the product WL⋯W1W_L\cdots W_1 shrinks or grows. Orthogonal or variance-preserving initialization (Var⁡(W)=1/n\operatorname{Var}(W)=1/n) keeps them stable.

Deep linear networks are also a standard object for theory. Saxe, McClelland and Ganguli (2014) showed that, although the function is linear, the training dynamics are not. Learning curves show plateaus followed by sudden drops, as each singular mode of the data is picked up in turn. The loss is non-convex in the factors even though it is convex in their product.

python
1import torch, torch.nn as nn
2
3# 1. Module: the identity, or simply no activation between layers
4reg = nn.Sequential(nn.Linear(16, 64), nn.ReLU(), nn.Linear(64, 1), nn.Identity()) # linear output for regression
5
6# 2. Functional / from scratch
7def linear_act(x, a=1.0):
8 return a * x # derivative is a everywhere
9
10# 3. Collapse: three linear layers equal one affine map
11torch.manual_seed(0)
12stack = nn.Sequential(nn.Linear(8, 32), nn.Linear(32, 32), nn.Linear(32, 4))
13A = stack[2].weight @ stack[1].weight @ stack[0].weight # 4 × 8
14c = stack[2].weight @ (stack[1].weight @ stack[0].bias + stack[1].bias) + stack[2].bias
15x = torch.randn(5, 8)
16print(torch.allclose(stack(x), x @ A.T + c, atol=1e-6)) # True
17print(torch.linalg.matrix_rank(A)) # at most 4
Linear activation in PyTorch: an identity module, a functional form, and a check that the stack collapses

Where it is used on purpose

Where it's used

Regression outputs

An unbounded target needs an unbounded output: the last layer is linear and trained with MSE, whose gradient at the output is simply y^−y\hat y-y.

Logits

The layer before a softmax or sigmoid is linear; the loss (cross-entropy with logits) supplies the nonlinearity.

Attention projections

Transformers compute Q=XWQQ=XW_Q, K=XWKK=XW_K, V=XWVV=XW_V and the output projection with no activation. In pre-norm Transformers (GPT-2 onward) each block adds its output to the residual stream, which is only ever summed into, never passed through an activation itself.

Linear bottlenecks

MobileNetV2 (Sandler et al., 2018) removes the ReLU from its narrow 1×1 projection layers, arguing that a nonlinearity in a low-dimensional space destroys information; the linear version was more accurate.

Low-rank factorization

LoRA (Hu et al., 2021) adds a product BABA of two thin linear maps to a frozen weight. The collapse is the point: BABA is one matrix of rank at most rr, with far fewer parameters (2⋅1024⋅64=131,0722\cdot1024\cdot64=131{,}072 versus 1,048,5761{,}048{,}576 for a full 1024×10241024\times1024).

Linear autoencoders

With linear layers and squared error, an autoencoder learns the PCA subspace (Baldi & Hornik, 1989).

Use when

  • +No saturation, no dead units: the gradient is a constant a
  • +Free to compute
  • +Right for unbounded outputs, logits, projections and residual streams
  • +Linear bottlenecks preserve information in narrow layers

Avoid when

  • −Adds no expressive power: any linear stack equals one layer
  • −Decision boundaries are hyperplanes only
  • −Unbounded activations: nothing limits their growth with depth
  • −Deep linear stacks still suffer vanishing/exploding products of weights

Check yourself

A network has 10 hidden layers of width 512, all with linear activations, and a softmax output. What can it represent?

Check yourself

Why do Transformer Q, K and V projections use no activation?

Remember

  • ✓f(x)=axf(x)=ax, f′(x)=af'(x)=a: no saturation, no dead units, no cost.
  • ✓Linear layers compose to one affine map: W2(W1x+b1)+b2=(W2W1)x+(W2b1+b2)W_2(W_1x+b_1)+b_2=(W_2W_1)x+(W_2b_1+b_2), with rank at most the narrowest width.
  • ✓A network without nonlinearities is a linear model however deep it is. One nonlinearity between layers breaks the collapse.
  • ✓Use it deliberately: regression outputs, logits, attention projections, the residual stream, MobileNetV2's linear bottlenecks, LoRA's low-rank product.
  • ✓Deep linear nets have non-convex, plateau-and-drop training dynamics (Saxe et al., 2014), even though the function they compute is linear.
← back to Deep Learning roadmap