Definition
The perceptron is one artificial neuron with a hard threshold, , plus a learning rule: whenever it gets an example wrong, add to the weights (and to the bias). Rosenblatt introduced it in 1957–58. It was the first neuron that learned its weights from data instead of having them set by hand.
The first trainable neuron
The McCulloch–Pitts neuron of 1943 already computed a weighted sum and fired above a threshold, but someone had to choose the weights. Rosenblatt's contribution was the update rule: show the neuron labelled examples and let its own mistakes move the weights.
Everything a modern network does with one unit is already here: a weighted sum, a bias, a nonlinearity, and weights changed by an error signal. What is missing is a smooth activation and a way to train more than one layer.
The neuron and its update rule
- input vector
- weights and bias, the only things that are learned
- pre-activation (score): positive means class +1, negative means −1
- the predicted label; the step function sign(·) is the activation
- the true label
- a mistake: the score has the wrong sign (or is exactly 0)
- learning rate; with w and b starting at 0 it only rescales them, so η = 1 is standard
Why this helps: write so the bias is just weight . After an update on , the new value of on that same point is . One update always pushes the offending point's score toward the correct sign. Correctly classified points are ignored.
The decision boundary is the set . In two dimensions that is a straight line with normal vector , and the bias shifts it away from the origin. Every update rotates or shifts that line, but it is always a single line.
Logic gates as a test bench
Boolean gates are the classic test: four inputs, two classes. Below, a 1 output is the label and a 0 output is .
| x₁ | x₂ | AND | OR | NAND | XOR |
|---|---|---|---|---|---|
| 0 | 0 | 0 | 0 | 1 | 0 |
| 0 | 1 | 0 | 1 | 1 | 1 |
| 1 | 0 | 0 | 1 | 1 | 1 |
| 1 | 1 | 1 | 1 | 0 | 0 |
Run the rule from , , visiting the rows in the order shown, with . Written as , OR stops at after 9 updates (6 epochs, the last one clean), i.e. the line . AND stops at after 18 updates (9 epochs). NAND is AND with every sign flipped and stops at . XOR never stops.
Worked example: one XOR epoch
| visit | x | y | s = w·x̃ | y·s | w after |
|---|---|---|---|---|---|
| 1 | (0, 0) | −1 | 0 | 0 → mistake | (−1, 0, 0) |
| 2 | (0, 1) | +1 | −1 | −1 → mistake | (0, 0, 1) |
| 3 | (1, 0) | +1 | 0 | 0 → mistake | (1, 1, 1) |
| 4 | (1, 1) | −1 | 3 | −3 → mistake | (0, 0, 0) |
The sum of the four updates is . With this visiting order the weights come back to exactly where they started, so epoch 2 is a copy of epoch 1, and so on forever. Other orders give different loops, but none ever ends, because no line separates XOR.
Why no line can do XOR
A four-line proof
- 1A separating line needs for , for , for and for .
- 2Add the middle two: , so .
- 3The first condition makes , so .
- 4That contradicts the last condition. No exists.
Geometrically: the two positive points sit on one diagonal of the square and the two negative points on the other. A half-plane is convex, so if it contains both ends of a diagonal it contains the midpoint . Both diagonals share that midpoint, so it would have to lie strictly on the positive side and strictly on the negative side at once.
Minsky and Papert's 1969 book Perceptrons made this kind of limit rigorous for single-layer perceptrons (XOR and parity are the simplest cases). Networks with a hidden layer could compute XOR, but there was no practical way to train their hidden weights. Interest and funding in neural networks fell sharply until backpropagation was popularised in 1986 (see the multilayer perceptron and backpropagation articles).
1import torch23X = torch.tensor([[0., 0.], [0., 1.], [1., 0.], [1., 1.]])4Xa = torch.cat([torch.ones(4, 1), X], dim=1) # x̃ = (1, x1, x2)56def train(y, epochs=20):7 w = torch.zeros(3) # (b, w1, w2)8 for epoch in range(1, epochs + 1):9 mistakes = 010 for xi, yi in zip(Xa, y):11 if yi * (w @ xi) <= 0: # wrong side, or on the line12 w += yi * xi13 mistakes += 114 if mistakes == 0:15 return w, epoch16 return w, None # never converged1718print(train(torch.tensor([-1., 1., 1., 1.]))) # OR : (tensor([-1., 2., 2.]), 6)19print(train(torch.tensor([-1., 1., 1., -1.]))) # XOR: (tensor([0., 0., 0.]), None)Check yourself
A perceptron has . Which gate does it compute on inputs in ?
Check yourself
Why can no single perceptron compute XOR, however long it trains?
Remember
- ✓Perceptron: ; on a mistake, and .
- ✓Each update moves the misclassified point's score toward the right sign by ‖x̃‖².
- ✓Its decision boundary is always a single straight line (a hyperplane in d dimensions).
- ✓AND, OR and NAND are linearly separable and are learned; XOR is not, and the rule cycles forever.
- ✓Fixing this needs a hidden layer, and training hidden layers needs differentiable activations.