Definition
AI, ML and deep learning are nested sets, not three competing technologies. Artificial intelligence is the goal: machines that do tasks we would call intelligent. Machine learning is one way to get there, where the behaviour is fitted from data instead of written by hand. Deep learning is the kind of machine learning that uses neural networks with many layers, and those layers learn the features as well as the final mapping.
Three ways to build the same program
Take one task: decide whether a photo contains a cat. The three families differ in which parts of the program a human writes and which parts are fitted to data.
Rule-based AI. An expert writes the logic directly, for example "if there are two pointy ears above two eyes and whiskers, say cat". This is AI with no learning at all. Expert systems, chess search and route planners all work this way. It breaks down when the rules are too many or too fuzzy to write down, and pixels are about as fuzzy as inputs get.
Classical machine learning. A human still designs the features: numbers computed from the raw input that are likely to matter, such as histograms of edge directions (HOG) or colour statistics. A learning algorithm (logistic regression, an SVM, boosted trees) then fits the mapping from features to label. The model can only be as good as the features it is given.
Deep learning. The network gets raw pixels and learns a stack of transformations. Trained vision networks consistently show early layers detecting edges and colour blobs, middle layers responding to textures and parts, and late layers responding to whole objects. Nobody programs that hierarchy; gradient descent finds it because it lowers the loss. This is representation learning, and it is what "deep" buys you.
- raw input (pixels, audio samples, token ids)
- hand-designed feature extractor, fixed before training
- learned model with parameters θ, applied to the features
- layer ℓ of the network, with its own learned parameters
- number of layers (the depth)
In classical ML only is learned and is frozen. In deep learning every is learned jointly from one loss, so the features are shaped by the task itself. That is what end-to-end training means.
When deep learning wins, and when it does not
Learning the features costs data and compute. A network with millions of free parameters has to discover structure that a human could have handed to a classical model for free. With little data, the hand-designed features win.
As data grows, hand-designed features become the ceiling: a classical model's accuracy plateaus at what its features allow. A large network keeps improving, because more data lets it learn better features. This is why deep learning took over images, audio and text, where raw inputs are high-dimensional and good features are hard to write down.
On tabular data (rows of meaningful columns such as age, income or number of purchases), the columns already are good features. Gradient-boosted trees remain the default there and often beat neural networks on medium-sized datasets (Grinsztajn et al., NeurIPS 2022).
| Model | What a human designs | Learned parameters |
|---|---|---|
| Rule-based digit reader | every rule | 0 |
| HOG features + linear SVM (pedestrian detector, 64×128 window) | the 3,780 HOG features | 3,781 (one weight per feature + bias) |
| Logistic regression on pixels | nothing; pixels go straight in | |
| MLP 784 → 256 → 10 | layer sizes only | |
| AlexNet (ImageNet, 2012) | architecture and loss | about 60 million |
A short history
| Year | Event | Family |
|---|---|---|
| 1956 | Dartmouth workshop coins "artificial intelligence" | AI |
| 1958 | Rosenblatt's perceptron: a single trainable neuron | ML / neural |
| 1959 | Samuel's self-improving checkers program; he coins "machine learning" | ML |
| 1969 | Minsky & Papert show a single-layer perceptron cannot compute XOR | neural |
| 1970s–80s | Expert systems (e.g. MYCIN): hand-written rules dominate | AI |
| 1986 | Rumelhart, Hinton & Williams popularize backpropagation for multilayer nets | neural |
| 1989–1998 | LeCun's convolutional networks read handwritten digits (LeNet-5, 1998) | DL |
| 1995–2001 | SVMs (1995) and random forests (2001): the classical-ML golden age | ML |
| 2006 | Hinton et al. train deep belief nets layer by layer; "deep learning" takes off as a name | DL |
| 2012 | AlexNet wins ImageNet: 15.3% top-5 error vs 26.2% for the runner-up | DL |
| 2017 | The transformer ("Attention Is All You Need") | DL |
| 2020s | Large language models trained on trillions of tokens | DL |
The neural-network idea is as old as the field. What changed around 2012 was scale: large labelled datasets (ImageNet has 1.2 million training images), GPUs that made training tens of times faster, and a few practical fixes such as ReLU activations and dropout.
The same task, two ways
1# Classical ML: a human chooses the features, the model learns the rest2from skimage.feature import hog3from sklearn.svm import LinearSVC45feats = [hog(img) for img in images] # phi(x): fixed, hand-designed6clf = LinearSVC().fit(feats, labels) # g_theta: learned78# Deep learning: raw pixels in, every layer learned from one loss9import torch.nn as nn1011net = nn.Sequential(12 nn.Conv2d(1, 16, 3), nn.ReLU(), # learns edge-like filters13 nn.Conv2d(16, 32, 3), nn.ReLU(), # learns combinations of edges14 nn.Flatten(), nn.LazyLinear(10), # learns the classifier15)Check yourself
A chess engine that searches the game tree with a hand-tuned evaluation function is…
Check yourself
What most precisely separates deep learning from classical machine learning?
Remember
- ✓AI ⊃ ML ⊃ DL: a goal, a method, and one family of that method.
- ✓Rule-based: a human writes everything. Classical ML: a human writes the features. DL: a human writes only the architecture and the loss.
- ✓Deep learning learns a hierarchy of features end to end from one loss.
- ✓DL wins with large, high-dimensional, perceptual data. Gradient-boosted trees still win on most small-to-medium tabular data.
- ✓The ideas date from the 1950s–80s; data, GPUs and a few training tricks made them work around 2012.