# Linear activation

> f(x) = ax passes the input through unchanged. Stacked linear layers collapse into one matrix, so it adds no expressive power, but it is right for outputs, projections and bottlenecks.

Source: https://www.mlfoundry.in/learn/linear-activation  
Section: Deep Learning › Activation Functions  
Level: core · 8 min read · updated 2026-10-11

**Linear activation:** (also **identity** activation, or "no activation") is $f(x)=ax$, usually with $a=1$. A layer with a linear activation computes an affine map $W x+b$ and nothing more.

Every other activation in this section exists because of one fact about this one: composing linear maps gives a linear map. A network with only linear activations, however deep and wide, computes exactly what a single linear layer computes. Depth adds parameters but no expressive power.

That does not make the linear activation useless. Wherever a layer should **not** bend its input, it is the correct choice: the output of a regression model, the logits fed to a softmax, the Q/K/V projections in attention, and the narrow bottleneck layers of MobileNetV2. "Linear activation" mostly means leaving the nonlinearity out on purpose.

## Formula and derivative

$$
f(x)=a\,x,\qquad f'(x)=a
$$

- $a$: slope; almost always 1 (the identity). Any other value can be absorbed into the next layer's weights
- $f'$: constant: every unit passes the gradient back scaled by exactly a, at every input

**Properties**

- **Range:** $(-\infty,\infty)$, unbounded
- **Monotonic:** yes for $a>0$
- **Smooth:** yes ($C^\infty$); $f''=0$
- **Zero-centred:** yes: odd function, f(0) = 0
- **Saturating:** never, so it neither vanishes nor bounds activations
- **Cost:** free: no operation at all for a = 1
- **Expressive power:** none added: any stack of linear layers equals one linear layer

> Figure: The identity and its constant derivative, with tanh for contrast: tanh agrees with the identity near 0 and bends away from it as |x| grows. For Gaussian inputs, the identity's output statistics are just the input's.

## The collapse of stacked linear layers

Two layers with linear activations, with the slope $a$ absorbed into the weights:

$$
W_2\big(W_1x+b_1\big)+b_2=\underbrace{(W_2W_1)}_{A}\,x+\underbrace{(W_2b_1+b_2)}_{c}
$$

- $A$: the product matrix; for $L$ layers $A=a^L\,W_L\cdots W_1$
- $c$: all the biases, pushed through the later layers

By induction, any number of layers collapses to one affine map $Ax+c$. Its decision boundaries are hyperplanes, the same ones logistic regression can draw. Its rank is at most the width of the narrowest layer: a $1024\to64\to1024$ linear stack is a rank-64 matrix. That is a constraint, not extra capacity.

> Figure: A grid and two rings of points pushed through L random 2-D layers h ← φ(Wh + b), animated layer by layer. With φ(x) = a·x, the output grid stays straight and evenly spaced, and it lands exactly on the dashed grid of one affine map Ax + c: at the default seed with L = 3 the largest gap is 6.0e−16, i.e. floating-point rounding. No line can separate the inner ring from the outer one before or after. Switch to tanh: the lines bend, and the same product of matrices misses the true output by up to 1.40.

## Worked example

Take $W_1=\begin{bmatrix}2&1\\-1&1\end{bmatrix}$, $b_1=(1,0)$, $W_2=\begin{bmatrix}1&-1\\0&2\end{bmatrix}$, $b_2=(0,1)$ and $x=(1,2)$:

Layer by layer versus the collapsed map, and the same weights with tanh in between (computed in Node)

| Quantity | Computation | Value |
| --- | --- | --- |
| $h_1$ | $W_1x+b_1=(2+2+1,\;-1+2)$ | $(5,\;1)$ |
| $y$ | $W_2h_1+b_2=(5-1,\;2+1)$ | $(4,\;3)$ |
| $A=W_2W_1$ | $\begin{bmatrix}1\cdot2+(-1)(-1) & 1-1\\ 0\cdot2+2(-1) & 0+2\end{bmatrix}$ | $\begin{bmatrix}3&0\\-2&2\end{bmatrix}$ |
| $c=W_2b_1+b_2$ | $(1,\;0)+(0,\;1)$ | $(1,\;1)$ |
| $Ax+c$ | $(3+1,\;-2+4+1)$ | $(4,\;3)$ — identical |
| with tanh after layer 1 | $W_2\tanh(h_1)+b_2$, $\tanh(h_1)=(0.9999,\,0.7616)$ | $(0.2383,\;2.5232)$ |

The second layer added parameters but nothing a single $2\times2$ matrix could not do. With tanh in between, the output changes completely, and no single matrix maps every $x$ to it.

## Effect on training

Gradients through a linear unit are simple: $f'=a$ everywhere, so there is no saturation and no dead region. A deep linear network still multiplies one weight matrix per layer in the backward pass, so its gradients can vanish or explode exactly as the product $W_L\cdots W_1$ shrinks or grows. Orthogonal or variance-preserving initialization ($\operatorname{Var}(W)=1/n$) keeps them stable.

Deep linear networks are also a standard object for **theory**. Saxe, McClelland and Ganguli (2014) showed that, although the function is linear, the training dynamics are not. Learning curves show plateaus followed by sudden drops, as each singular mode of the data is picked up in turn. The loss is non-convex in the factors even though it is convex in their product.

```python
import torch, torch.nn as nn

# 1. Module: the identity, or simply no activation between layers
reg = nn.Sequential(nn.Linear(16, 64), nn.ReLU(), nn.Linear(64, 1), nn.Identity())  # linear output for regression

# 2. Functional / from scratch
def linear_act(x, a=1.0):
    return a * x                                   # derivative is a everywhere

# 3. Collapse: three linear layers equal one affine map
torch.manual_seed(0)
stack = nn.Sequential(nn.Linear(8, 32), nn.Linear(32, 32), nn.Linear(32, 4))
A = stack[2].weight @ stack[1].weight @ stack[0].weight          # 4 × 8
c = stack[2].weight @ (stack[1].weight @ stack[0].bias + stack[1].bias) + stack[2].bias
x = torch.randn(5, 8)
print(torch.allclose(stack(x), x @ A.T + c, atol=1e-6))           # True
print(torch.linalg.matrix_rank(A))                                # at most 4
```

Linear activation in PyTorch: an identity module, a functional form, and a check that the stack collapses

## Where it is used on purpose

- **Regression outputs:** An unbounded target needs an unbounded output: the last layer is linear and trained with MSE, whose gradient at the output is simply $\hat y-y$.
- **Logits:** The layer before a softmax or sigmoid is linear; the loss (cross-entropy with logits) supplies the nonlinearity.
- **Attention projections:** Transformers compute $Q=XW_Q$, $K=XW_K$, $V=XW_V$ and the output projection with no activation. In pre-norm Transformers (GPT-2 onward) each block adds its output to the **residual stream**, which is only ever summed into, never passed through an activation itself.
- **Linear bottlenecks:** MobileNetV2 (Sandler et al., 2018) removes the ReLU from its narrow 1×1 projection layers, arguing that a nonlinearity in a low-dimensional space destroys information; the linear version was more accurate.
- **Low-rank factorization:** LoRA (Hu et al., 2021) adds a product $BA$ of two thin linear maps to a frozen weight. The collapse is the point: $BA$ is one matrix of rank at most $r$, with far fewer parameters ($2\cdot1024\cdot64=131{,}072$ versus $1{,}048{,}576$ for a full $1024\times1024$).
- **Linear autoencoders:** With linear layers and squared error, an autoencoder learns the PCA subspace (Baldi & Hornik, 1989).

> **Forgetting the activation:** `nn.Sequential(nn.Linear(784, 256), nn.Linear(256, 10))` is a bug that raises no error. It trains, and it reaches exactly the accuracy of multinomial logistic regression, because it **is** logistic regression with a rank-limited weight matrix. If adding layers does not change accuracy, check that there is a nonlinearity between them.

> **Why one nonlinearity is enough to escape:** The collapse needs every layer to be linear. Insert a single ReLU or tanh between two linear layers and you get a one-hidden-layer network, which can approximate any continuous function on a compact set given enough units (the universal approximation theorem). The figure shows a single tanh layer already bending the grid.

**Strengths**

- No saturation, no dead units: the gradient is a constant a
- Free to compute
- Right for unbounded outputs, logits, projections and residual streams
- Linear bottlenecks preserve information in narrow layers

**Limitations**

- Adds no expressive power: any linear stack equals one layer
- Decision boundaries are hyperplanes only
- Unbounded activations: nothing limits their growth with depth
- Deep linear stacks still suffer vanishing/exploding products of weights

**Question:** A network has 10 hidden layers of width 512, all with linear activations, and a softmax output. What can it represent?

- A. Any continuous function, given enough training
- B. Exactly what multinomial logistic regression can, with a weight matrix of rank at most 512
- C. Piecewise-linear functions with up to 2¹⁰ pieces
- D. Only the identity function

**Answer:** B. Exactly what multinomial logistic regression can, with a weight matrix of rank at most 512 — The 11 weight matrices multiply into one matrix $A$ (and the biases into one vector $c$), so the logits are $Ax+c$ and the softmax on top is multinomial logistic regression. The rank of $A$ is at most the smallest width, 512.

**Question:** Why do Transformer Q, K and V projections use no activation?

- A. Activations are too expensive at that scale
- B. Because the attention softmax and the MLP blocks already supply the nonlinearity; the projections only need to choose subspaces
- C. Linear layers cannot be trained otherwise
- D. ReLU would make attention scores negative

**Answer:** B. Because the attention softmax and the MLP blocks already supply the nonlinearity; the projections only need to choose subspaces — Attention is nonlinear through the softmax over $QK^\top$, and each block also has an MLP with a GELU or SwiGLU. The projections only rotate and select subspaces of the residual stream, which a linear map does exactly.

**Key points**

- $f(x)=ax$, $f'(x)=a$: no saturation, no dead units, no cost.
- Linear layers compose to one affine map: $W_2(W_1x+b_1)+b_2=(W_2W_1)x+(W_2b_1+b_2)$, with rank at most the narrowest width.
- A network without nonlinearities is a linear model however deep it is. One nonlinearity between layers breaks the collapse.
- Use it deliberately: regression outputs, logits, attention projections, the residual stream, MobileNetV2's linear bottlenecks, LoRA's low-rank product.
- Deep linear nets have non-convex, plateau-and-drop training dynamics (Saxe et al., 2014), even though the function they compute is linear.

---

From MLFoundry (https://www.mlfoundry.in). The web page adds interactive figures and diagrams.
