Definition
Fully convolutional network (FCN) is a network made only of convolution, pooling and upsampling layers, so it maps an image of any size to a score map of matching size. Long, Shelhamer and Darrell ("Fully Convolutional Networks for Semantic Segmentation", CVPR 2015) adapted ImageNet classifiers this way and trained them end to end for pixelwise Image segmentation, improving PASCAL VOC mean IU by about 20% relative to the previous best.
Before FCN, segmentation systems classified region proposals or superpixels with a CNN and merged the results (R-CNN-based SDS, Hariharan et al., 2014), or ran a patch classifier at every pixel. Both repeat almost the same computation for overlapping inputs, and both need pipelines of separate stages. FCN produces all pixel labels in one forward pass, about 175 ms per image against roughly 50 s for SDS.
Convolutionalising a classifier
A fully connected layer over a input is exactly a convolution with as many filters as outputs. So VGG16's fc6 becomes a conv with 4096 filters and fc7 a conv with 4096 filters, with the same weights. On a larger image the network no longer outputs one vector but a grid of them, one per -pixel step: it is the classifier slid over the image, with the shared convolutions computed once.
- side of the fc6 output map for an input (pool5 has stride 32, fc6 is a valid conv)
- score of class at cell : a convolution of the features , 21 outputs for PASCAL VOC (20 classes + background)
The released Caffe models pad the first convolution by 100 pixels and crop the output back to the input size, so small images still give a valid fc6 map. A "score" convolution then maps the 4096 channels to 21 class scores.
Upsampling and skip fusion
The stride-32 scores are coarse. FCN upsamples them with a Transposed convolution of stride and kernel size ( for , for ), initialised to bilinear interpolation. The final upsampler is kept fixed at bilinear; the intermediate ones are learned. Each kernel row is a tent:
- 1D bilinear kernel for even size ; the 2D kernel is
- 21-channel scores from conv7, pool4 and pool3 (the last two via new convs)
- upsampling by (transposed conv); is the FCN-8s output at input resolution
Deep layers know what is there but see it at stride 32. Shallow layers have finer resolution but less semantics. Summing their scores, coarse to fine, gives the "deep jet" of the paper. The networks are trained in stages, FCN-32s then 16s then 8s, each initialised from the previous one. Going below stride 8 gave diminishing returns.
Configuration and results (from the paper)
- Backbone
- VGG16 (ImageNet), fc6 → 7×7 conv, 4096 filters; fc7 → 1×1 conv, 4096; score → 1×1 conv, 21
- Upsamplers
- 2× steps: kernel 4, stride 2 (learned); final: kernel 64 / 32 / 16 at stride 32 / 16 / 8 (bilinear, fixed)
- Loss
- per-pixel multinomial logistic loss over the whole image (no patch sampling)
- VOC 2011 val subset
- mean IU 59.4 (FCN-32s), 62.4 (FCN-16s), 62.7 (FCN-8s)
- Test sets
- FCN-8s 62.7 mean IU on VOC 2011 test and 62.2 on VOC 2012 test, a 20% relative gain over the previous best (SDS, 52.6 on 2011)
The real gains were smaller (59.4 to 62.7) because learned scores are not oracle scores, and thin structures make up a small part of VOC pixels. The same resolution problem led to Dilated convolution in DeepLab and to the full decoders of U-Net.
Worked example: output size and a 2× upsample
| Input H × H | pool5 | fc6 / score map | Meaning |
|---|---|---|---|
| 224 × 224 | 7 × 7 | 1 × 1 | the original classifier: one prediction |
| 288 × 288 | 9 × 9 | 3 × 3 | classifier at 9 window positions, step 32 px |
| 384 × 384 | 12 × 12 | 6 × 6 | 36 windows, conv features computed once |
| 512 × 512 | 16 × 16 | 10 × 10 | 100 windows |
Upsample the score row (0, 1, 0.5) by 2 with the 4-tap bilinear kernel (0.25, 0.75, 0.75, 0.25), stride 2, padding 1
- 1Each input stamps a scaled kernel at offset 2i: 0 × kernel at 0–3, 1 × kernel at 2–5, 0.5 × kernel = (0.125, 0.375, 0.375, 0.125) at 4–7.
- 2Sum the stamps: (0, 0, 0.25, 0.75, 0.875, 0.625, 0.375, 0.125).
- 3Crop one value at each end (padding 1): (0, 0.25, 0.75, 0.875, 0.625, 0.375), twice the input length.
- 4This equals bilinear interpolation (align_corners = False) except at the last position: 0.375 instead of 0.5, because the transposed conv treats the outside as zero. Training then adjusts the 2× kernels freely.
1import numpy as np23def bilinear_kernel(size):4 """FCN's upsampling filter: kernel size 2f - f % 2 for factor f (4 for 2×, 64 for 32×)."""5 f = (size + 1) // 26 c = f - 1 if size % 2 else f - 0.57 k = 1 - np.abs(np.arange(size) - c) / f8 return np.outer(k, k)910def conv_transpose2d(x, w, stride, pad):11 """Single-channel transposed convolution: every input pixel stamps a scaled copy of w."""12 n, k = x.shape[0], w.shape[0]13 y = np.zeros(((n - 1) * stride + k,) * 2)14 for i in range(n):15 for j in range(n):16 y[i * stride:i * stride + k, j * stride:j * stride + k] += x[i, j] * w17 return y[pad:-pad, pad:-pad] if pad else y1819w = bilinear_kernel(4)20print(w[1]) # [0.1875 0.5625 0.5625 0.1875] = 0.75 * [.25 .75 .75 .25]21print(bilinear_kernel(64).shape) # (64, 64): the FCN-32s upsampler, stride 322223scores = np.array([[0.0, 1.0, 0.5]] * 3) # a 3×3 class-score map, constant down the columns24up = conv_transpose2d(scores, w, stride=2, pad=1)25print(up.shape, up[2].round(3)) # (6, 6) [0. 0.25 0.75 0.875 0.625 0.375]Check yourself
A 320 × 320 image goes through convolutionalised VGG16 with no extra padding. What size is the score map before upsampling?
Check yourself
What does FCN-16s add to FCN-32s?
Remember
- ✓fc layers = convolutions: fc6 → conv, fc7 → conv; any input size gives a score grid with stride 32.
- ✓Scores are upsampled by transposed convolutions initialised to bilinear (kernel 2f, stride f).
- ✓Skip fusion sums 1×1-conv scores from pool4 and pool3: .
- ✓Mean IU 59.4 → 62.4 → 62.7 for FCN-32s/16s/8s; 62.2 on VOC 2012 test; about 175 ms per image.
- ✓Trained end to end with per-pixel cross-entropy on whole images; the template for every later segmentation network.