Definition
VGGNet is the family of networks from Simonyan and Zisserman, "Very Deep Convolutional Networks for Large-Scale Image Recognition" (arXiv 2014, ICLR 2015), by Oxford's Visual Geometry Group. Every convolution is with stride 1 and padding 1, so it keeps the spatial size. Five stride-2 max-pools halve it, the width doubles after each pool from 64 to 512, and three dense layers (4096, 4096, 1000) finish the network. Configurations A to E differ only in depth, from 11 to 19 weight layers. D and E are the famous VGG-16 and VGG-19.
By 2014 the question after AlexNet and ZFNet was what actually matters in a CNN. The VGG paper isolated one variable, depth. It fixed every other design choice to the simplest option (smallest useful kernel, stride 1, same padding, one pooling type) and added layers. The result was second place in the ILSVRC-2014 classification task, with 7.3% top-5 test error from an ensemble of 7 nets, behind GoogLeNet (Inception). It was first in localisation, with 25.3% error. After the competition a 2-net ensemble of D and E reached 6.8%, and the best single net 7.0%.
VGG's real legacy is its uniformity. A VGG network is a list of numbers, like [64, 64, M, 128, 128, M, ...], and that made it the default backbone for detection, segmentation and style transfer for years. Its pretrained features still define the Perceptual loss used in super-resolution and image synthesis.
The key idea: stacks of 3×3 instead of big kernels
Two stacked convolutions (stride 1, no pooling between) have a receptive field, and three have . In general layers see . With channels in and out, the weights compare as:
- channels in and out of each layer (biases ignored)
- number of stacked layers; receptive field
- the single needs 81% more weights, and with stride 1 also 81% more multiply-adds
The stack also has ReLUs instead of one, so the function it computes is more non-linear and, in the paper's words, more discriminative. The paper reads the trade as a regulariser: a filter is forced to factor through pieces with non-linearities in between. The stack also weights its receptive field differently. The centre pixel reaches the output through paths and a corner through one, so the effective receptive field is concentrated in the middle.
The paper tested this directly. Network B, with pairs of layers, beat a version in which each pair was replaced by a single layer by 7% top-1 error on a centre crop. Configuration C adds convolutions to B: extra non-linearity at the same receptive field. C beat B, but D, which uses in those places, beat C. Depth helps, and so does spatial context.
The configurations
| config | weight layers | convs per block | params (lib) | paper / torchvision | G MACs | top-5 err. |
|---|---|---|---|---|---|---|
| A | 11 | 1-1-2-2-2 | 132,863,336 | 133M / vgg11 | 7.61 | 10.4% |
| A-LRN | 11 | A + LRN after conv1 | 132,863,336 | 133M / — | 7.61 | 10.5% |
| B | 13 | 2-2-2-2-2 | 133,047,848 | 133M / vgg13 | 11.31 | 9.9% |
| C | 16 | B + one 1×1 in blocks 3–5 | 133,638,952 | 134M / — | 11.77 | 9.4% |
| D (VGG-16) | 16 | 2-2-3-3-3 | 138,357,544 | 138M / vgg16 | 15.47 | 8.8% |
| E (VGG-19) | 19 | 2-2-4-4-4 | 143,667,240 | 144M / vgg19 | 19.63 | 9.0% |
Three things stand out. First, LRN (A-LRN) did not help, so B–E drop it. Second, adding 8 conv layers from A to E costs only 10.8M parameters, because the dense head (M in every configuration) dominates. Compute, by contrast, rises from 7.6 to 19.6 G multiply-adds. Third, depth saturated: with single-scale training E was no better than D. Training with scale jittering, where each image is rescaled so its short side is random in before cropping, brought D to 8.1% and E to 8.0% top-5.
Worked example: VGG-16's shapes, parameters and compute
- 1conv1_1: channels, parameters at , so M multiply-adds. conv1_2 (): parameters but G multiply-adds, 12% of the whole network.
- 2Each pool halves the side and the next block doubles the width: with 64, 128, 256, 512, 512 channels. A layer costs , so halving and doubling leaves it unchanged: conv1_2, conv2_2, conv3_2, conv3_3, conv4_2 and conv4_3 each cost exactly multiply-adds.
- 3After pool5: values. fc6 has parameters, more than all other layers together. fc7 adds M and fc8 M.
- 4Receptive fields grow by the current stride per layer: in block 1, then in block 2, and so on up to for conv5_3 and after pool5.
- 5Totals: conv parameters plus dense gives . Multiply-adds: G, of which G are in the convs.
At a glance
- Paper
- Simonyan & Zisserman, arXiv 1409.1556 (2014), ICLR 2015
- Building block
- conv, stride 1, pad 1, ReLU; max-pool, stride 2
- VGG-16
- 13 conv + 3 dense; 138,357,544 params; 15.47 G multiply-adds at 224×224
- Training
- SGD, batch 256, momentum 0.9, weight decay , dropout 0.5 in fc6/fc7; lr divided by 10 three times; 74 epochs; 2–3 weeks per net on 4 Titan Black GPUs
- Initialisation
- A was trained from random init; deeper nets started from A's first four conv layers and three dense layers. The authors later noted Glorot init works without this
- ILSVRC-2014
- 2nd in classification (7.3% top-5, 7 nets), 1st in localisation; 6.8% after the deadline with 2 nets
- torchvision
vgg16: 71.592% top-1, 90.382% top-5;vgg19: 72.376% / 90.876%;vgg16_bn: 73.36% top-1
At test time the paper converts the dense layers into convolutions: fc6 becomes a conv and fc7 and fc8 become convs. The network can then run on a whole, uncropped image of any size and produce a map of class scores, which is averaged spatially. This dense evaluation, the same idea as LeNet-5's C5, avoids recomputing overlapping crops.
1import torch, torch.nn as nn2from torchvision.models import vgg16, VGG16_Weights34CFG = {5 "A": [64, "M", 128, "M", 256, 256, "M", 512, 512, "M", 512, 512, "M"],6 "D": [64, 64, "M", 128, 128, "M", 256, 256, 256, "M", 512, 512, 512, "M", 512, 512, 512, "M"],7 "E": [64, 64, "M", 128, 128, "M", 256, 256, 256, 256, "M", 512, 512, 512, 512, "M", 512, 512, 512, 512, "M"],8}910def make_vgg(cfg, num_classes=1000):11 layers, c = [], 312 for v in cfg:13 if v == "M":14 layers.append(nn.MaxPool2d(2, 2))15 else:16 layers += [nn.Conv2d(c, v, 3, padding=1), nn.ReLU(inplace=True)]17 c = v18 return nn.Sequential(19 *layers, nn.Flatten(),20 nn.Linear(512 * 7 * 7, 4096), nn.ReLU(inplace=True), nn.Dropout(0.5),21 nn.Linear(4096, 4096), nn.ReLU(inplace=True), nn.Dropout(0.5),22 nn.Linear(4096, num_classes),23 )2425count = lambda m: sum(p.numel() for p in m.parameters())26print(count(make_vgg(CFG["D"]))) # 138357544, same as torchvision's vgg162728# three 3x3 layers vs one 7x7, C = 51229C = 51230stack = nn.Sequential(*[m for _ in range(3) for m in (nn.Conv2d(C, C, 3, padding=1, bias=False), nn.ReLU())])31big = nn.Conv2d(C, C, 7, padding=3, bias=False)32print(count(stack), count(big)) # 7077888 128450563334# frozen VGG-16 features for a perceptual loss (see "Perceptual loss")35feats = vgg16(weights=VGG16_Weights.IMAGENET1K_V1).features[:23].eval().requires_grad_(False)36TAPS = {3: "relu1_2", 8: "relu2_2", 15: "relu3_3", 22: "relu4_3"}3738def vgg_taps(x): # x: ImageNet-normalised images, N x 3 x H x W39 out = {}40 for i, layer in enumerate(feats):41 x = layer(x)42 if i in TAPS:43 out[TAPS[i]] = x44 return outUse when
- +Extremely simple and uniform: a whole network is one list of widths
- +Stacked 3×3 convs give large receptive fields with fewer weights and more non-linearity
- +Strong, transferable features; still the standard backbone for perceptual and style losses
- +Fully convolutional evaluation runs on images of any size
Avoid when
- −138–144M parameters, 74% of them in fc6: heavy to store and to fine-tune
- −15.5–19.6 G multiply-adds per 224×224 image, much more than GoogLeNet or ResNet-50 for lower accuracy
- −Large activation memory in the high-resolution early layers
- −No normalisation or shortcuts: depth beyond ~19 layers does not train well
Check yourself
Three stacked 3×3 conv layers (C channels in and out, no pooling between) are compared with one 7×7 layer. Which statement is correct?
Check yourself
Roughly what fraction of VGG-16's 138M parameters sit in the first fully connected layer (fc6)?
Remember
- ✓Only convs (stride 1, pad 1) and max-pools; widths 64→128→256→512→512; dense head 4096-4096-1000.
- ✓ stacked layers see pixels with weights instead of , and with ReLUs instead of one.
- ✓Configurations A–E: 11–19 weight layers, 133–144M parameters; D = VGG-16 (138.36M, 15.47 G multiply-adds), E = VGG-19.
- ✓fc6 holds 74% of VGG-16's parameters; the convs do 99% of the compute. Halving the side while doubling the width keeps each layer's cost constant.
- ✓ILSVRC-2014: 2nd in classification (7.3% top-5), 1st in localisation; scale jittering and dense evaluation were part of the recipe.
- ✓Its frozen features are the standard backbone for perceptual and style losses.