Definition
Linear activation (also identity activation, or "no activation") is , usually with . A layer with a linear activation computes an affine map and nothing more.
Every other activation in this section exists because of one fact about this one: composing linear maps gives a linear map. A network with only linear activations, however deep and wide, computes exactly what a single linear layer computes. Depth adds parameters but no expressive power.
That does not make the linear activation useless. Wherever a layer should not bend its input, it is the correct choice: the output of a regression model, the logits fed to a softmax, the Q/K/V projections in attention, and the narrow bottleneck layers of MobileNetV2. "Linear activation" mostly means leaving the nonlinearity out on purpose.
Formula and derivative
- slope; almost always 1 (the identity). Any other value can be absorbed into the next layer's weights
- constant: every unit passes the gradient back scaled by exactly a, at every input
Properties
- Range
- , unbounded
- Monotonic
- yes for
- Smooth
- yes ();
- Zero-centred
- yes: odd function, f(0) = 0
- Saturating
- never, so it neither vanishes nor bounds activations
- Cost
- free: no operation at all for a = 1
- Expressive power
- none added: any stack of linear layers equals one linear layer
The collapse of stacked linear layers
Two layers with linear activations, with the slope absorbed into the weights:
- the product matrix; for layers
- all the biases, pushed through the later layers
By induction, any number of layers collapses to one affine map . Its decision boundaries are hyperplanes, the same ones logistic regression can draw. Its rank is at most the width of the narrowest layer: a linear stack is a rank-64 matrix. That is a constraint, not extra capacity.
Worked example
Take , , , and :
| Quantity | Computation | Value |
|---|---|---|
| — identical | ||
| with tanh after layer 1 | , |
The second layer added parameters but nothing a single matrix could not do. With tanh in between, the output changes completely, and no single matrix maps every to it.
Effect on training
Gradients through a linear unit are simple: everywhere, so there is no saturation and no dead region. A deep linear network still multiplies one weight matrix per layer in the backward pass, so its gradients can vanish or explode exactly as the product shrinks or grows. Orthogonal or variance-preserving initialization () keeps them stable.
Deep linear networks are also a standard object for theory. Saxe, McClelland and Ganguli (2014) showed that, although the function is linear, the training dynamics are not. Learning curves show plateaus followed by sudden drops, as each singular mode of the data is picked up in turn. The loss is non-convex in the factors even though it is convex in their product.
1import torch, torch.nn as nn23# 1. Module: the identity, or simply no activation between layers4reg = nn.Sequential(nn.Linear(16, 64), nn.ReLU(), nn.Linear(64, 1), nn.Identity()) # linear output for regression56# 2. Functional / from scratch7def linear_act(x, a=1.0):8 return a * x # derivative is a everywhere910# 3. Collapse: three linear layers equal one affine map11torch.manual_seed(0)12stack = nn.Sequential(nn.Linear(8, 32), nn.Linear(32, 32), nn.Linear(32, 4))13A = stack[2].weight @ stack[1].weight @ stack[0].weight # 4 × 814c = stack[2].weight @ (stack[1].weight @ stack[0].bias + stack[1].bias) + stack[2].bias15x = torch.randn(5, 8)16print(torch.allclose(stack(x), x @ A.T + c, atol=1e-6)) # True17print(torch.linalg.matrix_rank(A)) # at most 4Where it is used on purpose
Where it's used
Regression outputs
An unbounded target needs an unbounded output: the last layer is linear and trained with MSE, whose gradient at the output is simply .
Logits
The layer before a softmax or sigmoid is linear; the loss (cross-entropy with logits) supplies the nonlinearity.
Attention projections
Transformers compute , , and the output projection with no activation. In pre-norm Transformers (GPT-2 onward) each block adds its output to the residual stream, which is only ever summed into, never passed through an activation itself.
Linear bottlenecks
MobileNetV2 (Sandler et al., 2018) removes the ReLU from its narrow 1×1 projection layers, arguing that a nonlinearity in a low-dimensional space destroys information; the linear version was more accurate.
Low-rank factorization
LoRA (Hu et al., 2021) adds a product of two thin linear maps to a frozen weight. The collapse is the point: is one matrix of rank at most , with far fewer parameters ( versus for a full ).
Linear autoencoders
With linear layers and squared error, an autoencoder learns the PCA subspace (Baldi & Hornik, 1989).
Use when
- +No saturation, no dead units: the gradient is a constant a
- +Free to compute
- +Right for unbounded outputs, logits, projections and residual streams
- +Linear bottlenecks preserve information in narrow layers
Avoid when
- −Adds no expressive power: any linear stack equals one layer
- −Decision boundaries are hyperplanes only
- −Unbounded activations: nothing limits their growth with depth
- −Deep linear stacks still suffer vanishing/exploding products of weights
Check yourself
A network has 10 hidden layers of width 512, all with linear activations, and a softmax output. What can it represent?
Check yourself
Why do Transformer Q, K and V projections use no activation?
Remember
- ✓, : no saturation, no dead units, no cost.
- ✓Linear layers compose to one affine map: , with rank at most the narrowest width.
- ✓A network without nonlinearities is a linear model however deep it is. One nonlinearity between layers breaks the collapse.
- ✓Use it deliberately: regression outputs, logits, attention projections, the residual stream, MobileNetV2's linear bottlenecks, LoRA's low-rank product.
- ✓Deep linear nets have non-convex, plateau-and-drop training dynamics (Saxe et al., 2014), even though the function they compute is linear.