# The perceptron

> A single neuron sign(w·x + b) that adds y·x to its weights on every mistake; it can only draw a straight line, so it cannot learn XOR.

Source: https://www.mlfoundry.in/learn/the-perceptron  
Section: Deep Learning › Deep Learning Fundamentals  
Level: core · 6 min read · updated 2026-10-09

**The perceptron:** is one artificial neuron with a hard threshold, $\hat y=\operatorname{sign}(w\cdot x+b)$, plus a learning rule: whenever it gets an example wrong, add $y\,x$ to the weights (and $y$ to the bias). Rosenblatt introduced it in 1957–58. It was the first neuron that learned its weights from data instead of having them set by hand.

## The first trainable neuron

The McCulloch–Pitts neuron of 1943 already computed a weighted sum and fired above a threshold, but someone had to choose the weights. Rosenblatt's contribution was the **update rule**: show the neuron labelled examples and let its own mistakes move the weights.

Everything a modern network does with one unit is already here: a weighted sum, a bias, a nonlinearity, and weights changed by an error signal. What is missing is a smooth activation and a way to train more than one layer.

## The neuron and its update rule

$$
\hat y=\operatorname{sign}(s),\qquad s=w\cdot x+b=w_1x_1+\dots+w_dx_d+b
$$

- $x\in\mathbb{R}^d$: input vector
- $w\in\mathbb{R}^d,\ b$: weights and bias, the only things that are learned
- $s$: pre-activation (score): positive means class +1, negative means −1
- $\hat y\in\{-1,+1\}$: the predicted label; the step function sign(·) is the activation

$$
\text{if } y\,s\le 0:\qquad w\leftarrow w+\eta\,y\,x,\qquad b\leftarrow b+\eta\,y
$$

- $y\in\{-1,+1\}$: the true label
- $y\,s\le0$: a mistake: the score has the wrong sign (or is exactly 0)
- $\eta>0$: learning rate; with w and b starting at 0 it only rescales them, so η = 1 is standard

Why this helps: write $\tilde x=(1,x)$ so the bias is just weight $w_0$. After an update on $(\tilde x,y)$, the new value of $y\,s$ on that same point is $y\,(w+y\tilde x)\cdot\tilde x=y\,s+\lVert\tilde x\rVert^2$. One update always pushes the offending point's score toward the correct sign. Correctly classified points are ignored.

The decision boundary is the set $w\cdot x+b=0$. In two dimensions that is a **straight line** with normal vector $w$, and the bias shifts it away from the origin. Every update rotates or shifts that line, but it is always a single line.

## Logic gates as a test bench

Boolean gates are the classic test: four inputs, two classes. Below, a 1 output is the label $+1$ and a 0 output is $-1$.

Truth tables. AND, OR and NAND each separate one corner of the square from the other three, which one line can do. XOR pairs opposite corners.

| x₁ | x₂ | AND | OR | NAND | XOR |
| --- | --- | --- | --- | --- | --- |
| 0 | 0 | 0 | 0 | 1 | 0 |
| 0 | 1 | 0 | 1 | 1 | 1 |
| 1 | 0 | 0 | 1 | 1 | 1 |
| 1 | 1 | 1 | 1 | 0 | 0 |

Run the rule from $w=0$, $b=0$, visiting the rows in the order shown, with $\eta=1$. Written as $(b,w_1,w_2)$, **OR** stops at $(-1,2,2)$ after 9 updates (6 epochs, the last one clean), i.e. the line $2x_1+2x_2=1$. **AND** stops at $(-4,3,2)$ after 18 updates (9 epochs). **NAND** is AND with every sign flipped and stops at $(4,-3,-2)$. **XOR** never stops.

> Figure: Pick a gate and press step (one input) or epoch (all four). The line is w·x + b = 0, the shaded side is predicted +1, and a red ring marks a misclassified point. OR and AND settle with zero mistakes in an epoch. Under XOR every epoch makes 4 mistakes and w returns to (0, 0, 0), so the mistake counter grows forever.

## Worked example: one XOR epoch

XOR from w = (b, w₁, w₂) = (0, 0, 0) with η = 1. Every visit is a mistake, and the four updates cancel.

| visit | x | y | s = w·x̃ | y·s | w after |
| --- | --- | --- | --- | --- | --- |
| 1 | (0, 0) | −1 | 0 | 0 → mistake | (−1, 0, 0) |
| 2 | (0, 1) | +1 | −1 | −1 → mistake | (0, 0, 1) |
| 3 | (1, 0) | +1 | 0 | 0 → mistake | (1, 1, 1) |
| 4 | (1, 1) | −1 | 3 | −3 → mistake | (0, 0, 0) |

The sum of the four updates is $\sum_i y_i\tilde x_i=-(1,0,0)+(1,0,1)+(1,1,0)-(1,1,1)=(0,0,0)$. With this visiting order the weights come back to exactly where they started, so epoch 2 is a copy of epoch 1, and so on forever. Other orders give different loops, but none ever ends, because no line separates XOR.

## Why no line can do XOR

**A four-line proof**

1. A separating line needs $b<0$ for $(0,0)$, $w_2+b>0$ for $(0,1)$, $w_1+b>0$ for $(1,0)$ and $w_1+w_2+b<0$ for $(1,1)$.
2. Add the middle two: $w_1+w_2+2b>0$, so $w_1+w_2+b>-b$.
3. The first condition makes $-b>0$, so $w_1+w_2+b>0$.
4. That contradicts the last condition. No $(w_1,w_2,b)$ exists.

Geometrically: the two positive points sit on one diagonal of the square and the two negative points on the other. A half-plane is convex, so if it contains both ends of a diagonal it contains the midpoint $(0.5,0.5)$. Both diagonals share that midpoint, so it would have to lie strictly on the positive side and strictly on the negative side at once.

Minsky and Papert's 1969 book **Perceptrons** made this kind of limit rigorous for single-layer perceptrons (XOR and parity are the simplest cases). Networks with a hidden layer could compute XOR, but there was no practical way to train their hidden weights. Interest and funding in neural networks fell sharply until backpropagation was popularised in 1986 (see the multilayer perceptron and backpropagation articles).

> **The rule is SGD in disguise:** Take the per-example loss $\ell=\max(0,-y\,s)$. When the point is wrong, $\nabla_w\ell=-y\,\tilde x$, so a gradient step is $w\leftarrow w+\eta\,y\,\tilde x$, which is the perceptron rule. When the point is right the gradient is 0 and nothing changes. Modern networks keep this "step against the loss gradient" idea and swap in smoother losses and activations.

> **The step function blocks deeper training:** $\operatorname{sign}$ has derivative 0 everywhere except at 0, where it is undefined. Stack two perceptrons and the output error cannot be passed back through the hidden step units, because the gradient is zero almost everywhere. Multilayer networks use differentiable activations (sigmoid, tanh, ReLU) for this reason.

```python
import torch

X = torch.tensor([[0., 0.], [0., 1.], [1., 0.], [1., 1.]])
Xa = torch.cat([torch.ones(4, 1), X], dim=1)        # x̃ = (1, x1, x2)

def train(y, epochs=20):
    w = torch.zeros(3)                               # (b, w1, w2)
    for epoch in range(1, epochs + 1):
        mistakes = 0
        for xi, yi in zip(Xa, y):
            if yi * (w @ xi) <= 0:                   # wrong side, or on the line
                w += yi * xi
                mistakes += 1
        if mistakes == 0:
            return w, epoch
    return w, None                                   # never converged

print(train(torch.tensor([-1., 1., 1., 1.])))   # OR : (tensor([-1., 2., 2.]), 6)
print(train(torch.tensor([-1., 1., 1., -1.])))  # XOR: (tensor([0., 0., 0.]), None)
```

The perceptron rule on the four gate inputs (bias folded in as x₀ = 1).

**Question:** A perceptron has $(b,w_1,w_2)=(-1,2,2)$. Which gate does it compute on inputs in $\{0,1\}^2$?

- A. AND
- B. OR
- C. NAND
- D. XOR

**Answer:** B. OR — The scores are $-1,1,1,3$ for $(0,0),(0,1),(1,0),(1,1)$, so only $(0,0)$ is negative. That is OR, the weights the rule finds in the figure.

**Question:** Why can no single perceptron compute XOR, however long it trains?

- A. The learning rate is too large, so it overshoots
- B. The step activation is not differentiable
- C. Its decision boundary is one straight line, and no line puts (0,1) and (1,0) on one side and (0,0) and (1,1) on the other
- D. Four examples are too few to learn from

**Answer:** C. Its decision boundary is one straight line, and no line puts (0,1) and (1,0) on one side and (0,0) and (1,1) on the other — The perceptron's prediction depends only on which side of the line $w\cdot x+b=0$ a point lies, and the four inequalities XOR needs contradict each other. The learning rate only rescales $w$ when it starts at 0, and differentiability is irrelevant to what a single line can represent.

**Key points**

- Perceptron: $\hat y=\operatorname{sign}(w\cdot x+b)$; on a mistake, $w\mathrel{+}=y\,x$ and $b\mathrel{+}=y$.
- Each update moves the misclassified point's score toward the right sign by ‖x̃‖².
- Its decision boundary is always a single straight line (a hyperplane in d dimensions).
- AND, OR and NAND are linearly separable and are learned; XOR is not, and the rule cycles forever.
- Fixing this needs a hidden layer, and training hidden layers needs differentiable activations.

---

From MLFoundry (https://www.mlfoundry.in). The web page adds interactive figures and diagrams.
