Definition
Binary step function (the Heaviside or threshold function) outputs when its input is at or above a threshold and otherwise: . A neuron fires or stays silent, with nothing in between.
The step is where artificial neurons began. McCulloch and Pitts (1943) modelled a neuron as an all-or-none threshold logic unit: sum the inputs and fire if the sum reaches a threshold. With the right weights such units compute AND, OR and NOT, so networks of them can implement any Boolean function.
Rosenblatt's perceptron (1958) kept the step output and added a learning rule. That rule is driven by mistakes, not by gradients. Widrow and Hoff's ADALINE (1960) used a step too, but trained on the linear sum before the step with least squares, because that quantity has a useful derivative. Every method for training through a step since then has made a similar workaround.
Formula and derivative
- pre-activation ; the bias moves the threshold to
- a convention: 1 here, sometimes 0 or ½ (torch.heaviside lets you choose)
- zero almost everywhere; as a distribution it is the Dirac delta
Two common variants: the sign function , which is and is used by the perceptron and by binarized networks, and a step with a threshold , . All of them are piecewise constant, so their derivatives are zero wherever they exist.
Properties
- Range
- (sign: )
- Monotonic
- yes, non-decreasing
- Continuous / smooth
- neither: a jump of 1 at
- Derivative
- 0 for , undefined at 0: no gradient signal
- Zero-centred
- no (outputs 0 or 1); sign is
- Saturating
- everywhere: the output never changes away from the jump
- Cost
- one comparison; outputs fit in 1 bit
Why gradient descent gets nothing
Take a unit inside any loss . The chain rule gives . The middle factor is 0 for every input not exactly at the threshold. So every weight feeding a step unit gets gradient 0, and so does everything below it in the network.
The loss is not constant: moving the threshold past a data point changes a prediction. But as a function of the parameters it is piecewise flat. It is a staircase that jumps at the points where some example changes side, and it is flat in between. Gradient descent reads the slope where it stands, which is zero, and stays put.
Workarounds: surrogates and straight-through
Replace the step by a smooth approximation. tends to as and has slope at 0, so it gives a gradient everywhere. This is how the field moved from threshold units to sigmoid units in the 1980s. The trade-off is in : a steep surrogate is faithful to the step but its gradient is nearly zero away from the threshold. At the slope at is ; at it is .
Keep the step, fake its derivative. The straight-through estimator (STE) uses in the forward pass but replaces in the backward pass by something useful: (identity), or , the derivative of hard-tanh. Bengio, Léonard and Courville (2013) analysed it and credit the idea to Hinton's 2012 lectures. The gradient is biased, because it is not the gradient of any loss the network actually computes, but it works well in practice.
- the hard-tanh STE used by Binarized Neural Networks; the plain STE uses 1
- the gradient arriving from above, passed straight through
Worked example
| x | H(x) | H′(x) | σ(2x) | d/dx σ(2x) | STE ∂y/∂x |
|---|---|---|---|---|---|
| −2 | 0 | 0 | 0.0180 | 0.0353 | 0 |
| −0.5 | 0 | 0 | 0.2689 | 0.3932 | 1 |
| −0.1 | 0 | 0 | 0.4502 | 0.4950 | 1 |
| 0 | 1 | undefined | 0.5000 | 0.5000 | 1 |
| 0.1 | 1 | 0 | 0.5498 | 0.4950 | 1 |
| 0.5 | 1 | 0 | 0.7311 | 0.3932 | 1 |
| 2 | 1 | 0 | 0.9820 | 0.0353 | 0 |
The middle column is the problem in one line. The surrogate's slope peaks at and is still at . The STE is 1 inside the window and 0 outside, so units far from the threshold are treated as saturated.
1import torch23x = torch.tensor([-1.5, -0.2, 0.0, 0.3, 2.0], requires_grad=True)45# 1. Built-in: comparisons and torch.heaviside produce no useful gradient6y = torch.heaviside(x.detach(), torch.tensor(1.0)) # H(0) = 17print(y) # tensor([0., 0., 1., 1., 1.])8print((x >= 0).float().requires_grad) # False: the graph is cut910# 2. Straight-through in one line: forward = step, backward = identity11y = x + ((x >= 0).float() - x).detach()12y.sum().backward()13print(x.grad) # tensor([1., 1., 1., 1., 1.])1415# 3. From scratch: hard-tanh STE, as in Binarized Neural Networks16class StepSTE(torch.autograd.Function):17 @staticmethod18 def forward(ctx, x):19 ctx.save_for_backward(x)20 return (x >= 0).to(x.dtype)21 @staticmethod22 def backward(ctx, g):23 (x,) = ctx.saved_tensors24 return g * (x.abs() <= 1).to(g.dtype) # pass the gradient only near the threshold2526x.grad = None27StepSTE.apply(x).sum().backward()28print(x.grad) # tensor([0., 1., 1., 1., 0.])Where steps are still used
Where it's used
Binarized and quantized networks
BinaryConnect (2015) and Binarized Neural Networks (Hubara et al., 2016) use sign(·) for weights and activations, trained with STE, so matrix products become XNOR and popcount operations.
Discrete latent codes
VQ-VAE (van den Oord et al., 2017) passes gradients straight through its non-differentiable nearest-codebook lookup.
Spiking neural networks
A spike is a step of the membrane potential; surrogate gradients replace its derivative during training.
Decision outputs
Any classifier's final decision is a step, applied after training at inference time.
Use when
- +Exact binary output: interpretable on/off, 1-bit storage
- +Cheapest possible activation: one comparison
- +Binarized networks replace multiply-adds with bit operations
Avoid when
- −Derivative is 0 almost everywhere: no gradient descent without a surrogate or STE
- −Not continuous: tiny input changes can flip the output
- −STE gradients are biased and can stall (as in the figure)
- −Not zero-centred (0/1 version)
Check yourself
A hidden layer uses H(x) and the network is trained with plain backprop. What gradient do the weights below that layer receive?
Check yourself
You replace H(x) by σ(kx) during training. What happens as k grows very large?
Remember
- ✓: the all-or-none neuron of McCulloch & Pitts (1943) and Rosenblatt's perceptron.
- ✓ for (undefined at 0), so backprop sends zero gradient through it: the loss is a staircase in the weights.
- ✓Smooth surrogates restore a gradient. Steeper ones are more faithful to the step but have less gradient away from the threshold.
- ✓The straight-through estimator keeps forward and uses or backward. It is biased but powers binarized nets and VQ-VAE.
- ✓Use steps for final decisions or binarized inference, not as trainable hidden activations.