Binary step function

The original all-or-none neuron: output 1 if the input crosses a threshold, else 0. Its derivative is 0 everywhere it exists, so gradient descent gets no signal through it.

8 min read691 wordscore4 viewsupdated 2026-10-11

Definition

Binary step function (the Heaviside or threshold function) outputs 11 when its input is at or above a threshold and 00 otherwise: H(x)=1[x≥0]H(x)=\mathbb{1}[x\ge0]. A neuron H(w⊤x+b)H(w^\top x+b) fires or stays silent, with nothing in between.

The step is where artificial neurons began. McCulloch and Pitts (1943) modelled a neuron as an all-or-none threshold logic unit: sum the inputs and fire if the sum reaches a threshold. With the right weights such units compute AND, OR and NOT, so networks of them can implement any Boolean function.

Rosenblatt's perceptron (1958) kept the step output and added a learning rule. That rule is driven by mistakes, not by gradients. Widrow and Hoff's ADALINE (1960) used a step too, but trained on the linear sum before the step with least squares, because that quantity has a useful derivative. Every method for training through a step since then has made a similar workaround.

Formula and derivative

H(x)={1x≥00x<0H′(x)={0x≠0undefinedx=0H(x)=\begin{cases}1 & x\ge 0\\ 0 & x<0\end{cases}\qquad\qquad H'(x)=\begin{cases}0 & x\ne 0\\ \text{undefined} & x=0\end{cases}
xx
pre-activation w⊤x+bw^\top x+b; the bias bb moves the threshold to −b-b
H(0)H(0)
a convention: 1 here, sometimes 0 or ½ (torch.heaviside lets you choose)
H′H'
zero almost everywhere; as a distribution it is the Dirac delta δ(x)\delta(x)

Two common variants: the sign function sign⁡(x)∈{−1,+1}\operatorname{sign}(x)\in\{-1,+1\}, which is 2H(x)−12H(x)-1 and is used by the perceptron and by binarized networks, and a step with a threshold θ\theta, H(x−θ)H(x-\theta). All of them are piecewise constant, so their derivatives are zero wherever they exist.

Properties

Range
{0,1}\{0,1\} (sign: {−1,+1}\{-1,+1\})
Monotonic
yes, non-decreasing
Continuous / smooth
neither: a jump of 1 at x=0x=0
Derivative
0 for x≠0x\ne0, undefined at 0: no gradient signal
Zero-centred
no (outputs 0 or 1); sign is
Saturating
everywhere: the output never changes away from the jump
Cost
one comparison; outputs fit in 1 bit
f(x) on x ∈ [−5, 5] · drag on the plot to move the probex = 1.00 · f(x) = 1.000 · f′(x) = 0.000−0.200.000.200.400.600.801.001.20−4−2024x →Binary stepBinary step f′Sigmoid
activation
compare
range {0, 1}monotonic yessmooth nozero-centred no
Threshold at 0 with H(0) = 1. The derivative is 0 everywhere except x = 0, where it is undefined (a Dirac delta), so gradients carry no signal.
signal through the nonlinearity · histogram of f(x) for x ~ N(0.0, 1.00²)exact quadrature, no samplingE[f]0.000.501.00output value f(x) → (axis spans the central ±4σ of inputs)
E[f(x)] 0.500Var[f(x)] 0.250E[f′(x)] 0.000f′ = 0 for 100.0% of inputsE[f′]¹⁰ 0
E[f′]¹⁰ is a rough heuristic for the gradient scale after 10 such layers: it ignores the weights and assumes every layer sees this same input distribution.
fig.The step and its derivative, which is flat zero apart from the jump at 0. The sigmoid is overlaid for comparison: it is a smoothed step whose slope is never exactly zero.

Why gradient descent gets nothing

Take a unit p=H(wx+b)p=H(wx+b) inside any loss L\mathcal{L}. The chain rule gives ∂L/∂w=∂L/∂p⋅H′(wx+b)⋅x\partial\mathcal{L}/\partial w=\partial\mathcal{L}/\partial p\cdot H'(wx+b)\cdot x. The middle factor is 0 for every input not exactly at the threshold. So every weight feeding a step unit gets gradient 0, and so does everything below it in the network.

The loss is not constant: moving the threshold past a data point changes a prediction. But as a function of the parameters it is piecewise flat. It is a staircase that jumps at the points where some example changes side, and it is flat in between. Gradient descent reads the slope where it stands, which is zero, and stays put.

model p = H(x − θ) on 24 points · θ = 2.20000.51y = 1 ● · y = 0 ● · ring = misclassified−3−2−10123loss vs threshold θ · step: MSE = error rate · sigmoid MSE (k = 2.0)00.20.40.6best error 0.125−3−2−10123θstep 0/60 · dL/dθ = 0 · error rate 0.500
train with
GD, lr = 1θ after 60 steps: 2.200, error 0.500
fig.A one-parameter classifier p = g(x − θ) on 24 points, trained by gradient descent (lr = 1) on the mean squared error from θ₀ = 2.2. With the true step derivative, dL/dθ is exactly 0 and θ never moves (error stays 0.5). The sigmoid surrogate σ(2(x − θ)) reaches the best error, 0.125, by step 9 and ends at θ = −0.036. The straight-through estimator keeps the step in the forward pass and moves θ too, but stalls after 7 steps at error 0.167, when the errors inside its window cancel. Push k to 30: the surrogate is nearly a step again, the starting gradient falls to 1.2e−3, and after 60 steps the error is still 0.333.

Workarounds: surrogates and straight-through

Replace the step by a smooth approximation. σ(kx)\sigma(kx) tends to H(x)H(x) as k→∞k\to\infty and has slope k/4k/4 at 0, so it gives a gradient everywhere. This is how the field moved from threshold units to sigmoid units in the 1980s. The trade-off is in kk: a steep surrogate is faithful to the step but its gradient is nearly zero away from the threshold. At k=10k=10 the slope at x=0.5x=0.5 is 0.0660.066; at k=100k=100 it is 1.9×10−201.9\times10^{-20}.

Keep the step, fake its derivative. The straight-through estimator (STE) uses HH in the forward pass but replaces H′H' in the backward pass by something useful: 11 (identity), or 1[∣x∣≤1]\mathbb{1}[|x|\le1], the derivative of hard-tanh. Bengio, Léonard and Courville (2013) analysed it and credit the idea to Hinton's 2012 lectures. The gradient is biased, because it is not the gradient of any loss the network actually computes, but it works well in practice.

forward: y=H(x)backward: ∂L∂x≈∂L∂y⋅1[∣x∣≤1]\text{forward: } y=H(x)\qquad\text{backward: } \frac{\partial\mathcal{L}}{\partial x}\approx\frac{\partial\mathcal{L}}{\partial y}\cdot\mathbb{1}\big[|x|\le1\big]
1[∣x∣≤1]\mathbb{1}[|x|\le1]
the hard-tanh STE used by Binarized Neural Networks; the plain STE uses 1
∂L/∂y\partial\mathcal{L}/\partial y
the gradient arriving from above, passed straight through

Worked example

xH(x)H′(x)σ(2x)d/dx σ(2x)STE ∂y/∂x
−2000.01800.03530
−0.5000.26890.39321
−0.1000.45020.49501
01undefined0.50000.50001
0.1100.54980.49501
0.5100.73110.39321
2100.98200.03530
The step, its true derivative, a sigmoid surrogate σ(2x) and the hard-tanh STE (computed in Node)

The middle column is the problem in one line. The surrogate's slope peaks at k/4=0.5k/4=0.5 and is still 0.0350.035 at ∣x∣=2|x|=2. The STE is 1 inside the window and 0 outside, so units far from the threshold are treated as saturated.

python
1import torch
2
3x = torch.tensor([-1.5, -0.2, 0.0, 0.3, 2.0], requires_grad=True)
4
5# 1. Built-in: comparisons and torch.heaviside produce no useful gradient
6y = torch.heaviside(x.detach(), torch.tensor(1.0)) # H(0) = 1
7print(y) # tensor([0., 0., 1., 1., 1.])
8print((x >= 0).float().requires_grad) # False: the graph is cut
9
10# 2. Straight-through in one line: forward = step, backward = identity
11y = x + ((x >= 0).float() - x).detach()
12y.sum().backward()
13print(x.grad) # tensor([1., 1., 1., 1., 1.])
14
15# 3. From scratch: hard-tanh STE, as in Binarized Neural Networks
16class StepSTE(torch.autograd.Function):
17 @staticmethod
18 def forward(ctx, x):
19 ctx.save_for_backward(x)
20 return (x >= 0).to(x.dtype)
21 @staticmethod
22 def backward(ctx, g):
23 (x,) = ctx.saved_tensors
24 return g * (x.abs() <= 1).to(g.dtype) # pass the gradient only near the threshold
25
26x.grad = None
27StepSTE.apply(x).sum().backward()
28print(x.grad) # tensor([0., 1., 1., 1., 0.])
Steps in PyTorch: no gradient by default, a straight-through trick, and a custom autograd function

Where steps are still used

Where it's used

Binarized and quantized networks

BinaryConnect (2015) and Binarized Neural Networks (Hubara et al., 2016) use sign(·) for weights and activations, trained with STE, so matrix products become XNOR and popcount operations.

Discrete latent codes

VQ-VAE (van den Oord et al., 2017) passes gradients straight through its non-differentiable nearest-codebook lookup.

Spiking neural networks

A spike is a step of the membrane potential; surrogate gradients replace its derivative during training.

Decision outputs

Any classifier's final decision 1[p≥0.5]\mathbb{1}[p\ge0.5] is a step, applied after training at inference time.

Use when

  • +Exact binary output: interpretable on/off, 1-bit storage
  • +Cheapest possible activation: one comparison
  • +Binarized networks replace multiply-adds with bit operations

Avoid when

  • −Derivative is 0 almost everywhere: no gradient descent without a surrogate or STE
  • −Not continuous: tiny input changes can flip the output
  • −STE gradients are biased and can stall (as in the figure)
  • −Not zero-centred (0/1 version)

Check yourself

A hidden layer uses H(x) and the network is trained with plain backprop. What gradient do the weights below that layer receive?

Check yourself

You replace H(x) by σ(kx) during training. What happens as k grows very large?

Remember

  • ✓H(x)=1[x≥0]H(x)=\mathbb{1}[x\ge0]: the all-or-none neuron of McCulloch & Pitts (1943) and Rosenblatt's perceptron.
  • ✓H′(x)=0H'(x)=0 for x≠0x\ne0 (undefined at 0), so backprop sends zero gradient through it: the loss is a staircase in the weights.
  • ✓Smooth surrogates σ(kx)\sigma(kx) restore a gradient. Steeper ones are more faithful to the step but have less gradient away from the threshold.
  • ✓The straight-through estimator keeps HH forward and uses 11 or 1[∣x∣≤1]\mathbb{1}[|x|\le1] backward. It is biased but powers binarized nets and VQ-VAE.
  • ✓Use steps for final decisions or binarized inference, not as trainable hidden activations.
← back to Deep Learning roadmap