FCN

FCN (Long, Shelhamer, Darrell, 2015) turns a classifier's fc layers into convolutions, predicts class scores at stride 32, upsamples them with transposed convolutions and adds pool4 and pool3 scores (FCN-8s): 62.2 mIoU on VOC 2012.

7 min read656 wordscore0 viewsupdated 2026-10-15

Definition

Fully convolutional network (FCN) is a network made only of convolution, pooling and upsampling layers, so it maps an image of any size to a score map of matching size. Long, Shelhamer and Darrell ("Fully Convolutional Networks for Semantic Segmentation", CVPR 2015) adapted ImageNet classifiers this way and trained them end to end for pixelwise Image segmentation, improving PASCAL VOC mean IU by about 20% relative to the previous best.

Before FCN, segmentation systems classified region proposals or superpixels with a CNN and merged the results (R-CNN-based SDS, Hariharan et al., 2014), or ran a patch classifier at every pixel. Both repeat almost the same computation for overlapping inputs, and both need pipelines of separate stages. FCN produces all pixel labels in one forward pass, about 175 ms per image against roughly 50 s for SDS.

Convolutionalising a classifier

A fully connected layer over a 7×7×5127\times7\times512 input is exactly a 7×77\times7 convolution with as many filters as outputs. So VGG16's fc6 becomes a 7×77\times7 conv with 4096 filters and fc7 a 1×11\times1 conv with 4096 filters, with the same weights. On a larger image the network no longer outputs one vector but a grid of them, one per 3232-pixel step: it is the classifier slid over the image, with the shared convolutions computed once.

hout=⌊H32⌋−7+1,sk(i,j)=∑cwk,c fc(i,j)+bkh_{\text{out}}=\Big\lfloor\frac{H}{32}\Big\rfloor-7+1,\qquad s_{k}(i,j)=\sum_{c}w_{k,c}\,f_c(i,j)+b_k
houth_{\text{out}}
side of the fc6 output map for an H×HH\times H input (pool5 has stride 32, fc6 is a valid 7×77\times7 conv)
sk(i,j)s_k(i,j)
score of class kk at cell (i,j)(i,j): a 1×11\times1 convolution of the features ff, 21 outputs for PASCAL VOC (20 classes + background)

The released Caffe models pad the first convolution by 100 pixels and crop the output back to the input size, so small images still give a valid fc6 map. A 1×11\times1 "score" convolution then maps the 4096 channels to 21 class scores.

FCN-VGG16: classifier → fully convolutional, then upsample and fuse skipsimageH×W3pool1/264pool2/4128pool3/8256pool4/16512pool5/32512fc6-7/3240967×7, 1×1convsscore21×32 up → FCN-32s59.4 mean IU2× deconv(bilinear init)1×1 conv → 21+×16 up → FCN-16s62.4 mean IU2× deconv1×1 conv → 21+×8 up → FCN-8s62.7 mean IU21 score channels = 20 VOC classes + background · skip scores are summed, not concatenatedoutput 21×H×W logits, per-pixel softmax loss · mean IU on a PASCAL VOC 2011 val subsetone forward pass for the whole image, any input size: no patches, no region proposals
fig.FCN-VGG16. The convolutionalised classifier gives 21 class scores at stride 32. FCN-32s upsamples them ×32 in one step. FCN-16s first upsamples them ×2 with a learnable transposed convolution and adds 1×1-conv scores computed from pool4 (stride 16); FCN-8s repeats that with pool3 (stride 8), then upsamples ×8. Mean IU on a PASCAL VOC 2011 validation subset rises from 59.4 to 62.4 to 62.7.

Upsampling and skip fusion

The stride-32 scores are coarse. FCN upsamples them with a Transposed convolution of stride ff and kernel size 2f2f (6464 for f=32f=32, 44 for f=2f=2), initialised to bilinear interpolation. The final upsampler is kept fixed at bilinear; the intermediate 2×2\times ones are learned. Each kernel row is a tent:

w(t)=1−∣t−c∣f,c=f−12, t=0,…,2f−1;z=U8(U2(U2(s32)+s16)+s8)w(t)=1-\frac{|t-c|}{f},\quad c=f-\tfrac12,\ t=0,\dots,2f-1;\qquad z=U_{8}\Big(U_{2}\big(U_{2}(s_{32})+s_{16}\big)+s_{8}\Big)
w(t)w(t)
1D bilinear kernel for even size 2f2f; the 2D kernel is w(t) w(t′)w(t)\,w(t')
s32, s16, s8s_{32},\,s_{16},\,s_{8}
21-channel scores from conv7, pool4 and pool3 (the last two via new 1×11\times1 convs)
UfU_f
upsampling by ff (transposed conv); zz is the FCN-8s output at input resolution

Deep layers know what is there but see it at stride 32. Shallow layers have finer resolution but less semantics. Summing their scores, coarse to fine, gives the "deep jet" of the paper. The networks are trained in stages, FCN-32s then 16s then 8s, each initialised from the previous one. Going below stride 8 gave diminishing returns.

Configuration and results (from the paper)

Backbone
VGG16 (ImageNet), fc6 → 7×7 conv, 4096 filters; fc7 → 1×1 conv, 4096; score → 1×1 conv, 21
Upsamplers
2× steps: kernel 4, stride 2 (learned); final: kernel 64 / 32 / 16 at stride 32 / 16 / 8 (bilinear, fixed)
Loss
per-pixel multinomial logistic loss over the whole image (no patch sampling)
VOC 2011 val subset
mean IU 59.4 (FCN-32s), 62.4 (FCN-16s), 62.7 (FCN-8s)
Test sets
FCN-8s 62.7 mean IU on VOC 2011 test and 62.2 on VOC 2012 test, a 20% relative gain over the previous best (SDS, 52.6 on 2011)
FCN-32s: conv7 scores, one ×32 upsampleoracle object score per cell, stride 32 (4×4 grid)bilinear ×32 → 128×128, threshold 0.50.0900.020.070.160.230.320.120.160.830.900.050.11000correct 1689 · missed 1439 · false + 5 pxdashed = ground truth (pole is 5 px wide)IoU object0.539IoU background0.902mean IoU0.720pole pixels found0.000a 5-px pole needs a cell that is more than half pole: impossible at stride 16 (5/16) or 32
variant
upsampling
fig.An oracle FCN: each coarse cell holds the true object fraction, the best a stride-s score map can do. Default FCN-32s (4×4 cells, bilinear ×32): object IoU 0.539, mean IoU 0.720, and the 5-pixel pole is invisible because its cells score at most 0.16. FCN-16s: object IoU 0.694. FCN-8s: object IoU 0.904, mean IoU 0.941, and 70.4% of the pole pixels are found. Switch FCN-8s to nearest upsampling and object IoU falls to 0.773: blocky edges add 441 false-positive pixels.

The real gains were smaller (59.4 to 62.7) because learned scores are not oracle scores, and thin structures make up a small part of VOC pixels. The same resolution problem led to Dilated convolution in DeepLab and to the full decoders of U-Net.

Worked example: output size and a 2× upsample

Input H × Hpool5fc6 / score mapMeaning
224 × 2247 × 71 × 1the original classifier: one prediction
288 × 2889 × 93 × 3classifier at 9 window positions, step 32 px
384 × 38412 × 126 × 636 windows, conv features computed once
512 × 51216 × 1610 × 10100 windows
VGG16 made fully convolutional, without the 100-pixel padding: pool5 = H / 32, fc6 output = pool5 − 6

Upsample the score row (0, 1, 0.5) by 2 with the 4-tap bilinear kernel (0.25, 0.75, 0.75, 0.25), stride 2, padding 1

  1. 1Each input stamps a scaled kernel at offset 2i: 0 × kernel at 0–3, 1 × kernel at 2–5, 0.5 × kernel = (0.125, 0.375, 0.375, 0.125) at 4–7.
  2. 2Sum the stamps: (0, 0, 0.25, 0.75, 0.875, 0.625, 0.375, 0.125).
  3. 3Crop one value at each end (padding 1): (0, 0.25, 0.75, 0.875, 0.625, 0.375), twice the input length.
  4. 4This equals bilinear interpolation (align_corners = False) except at the last position: 0.375 instead of 0.5, because the transposed conv treats the outside as zero. Training then adjusts the 2× kernels freely.
python
1import numpy as np
2
3def bilinear_kernel(size):
4 """FCN's upsampling filter: kernel size 2f - f % 2 for factor f (4 for 2×, 64 for 32×)."""
5 f = (size + 1) // 2
6 c = f - 1 if size % 2 else f - 0.5
7 k = 1 - np.abs(np.arange(size) - c) / f
8 return np.outer(k, k)
9
10def conv_transpose2d(x, w, stride, pad):
11 """Single-channel transposed convolution: every input pixel stamps a scaled copy of w."""
12 n, k = x.shape[0], w.shape[0]
13 y = np.zeros(((n - 1) * stride + k,) * 2)
14 for i in range(n):
15 for j in range(n):
16 y[i * stride:i * stride + k, j * stride:j * stride + k] += x[i, j] * w
17 return y[pad:-pad, pad:-pad] if pad else y
18
19w = bilinear_kernel(4)
20print(w[1]) # [0.1875 0.5625 0.5625 0.1875] = 0.75 * [.25 .75 .75 .25]
21print(bilinear_kernel(64).shape) # (64, 64): the FCN-32s upsampler, stride 32
22
23scores = np.array([[0.0, 1.0, 0.5]] * 3) # a 3×3 class-score map, constant down the columns
24up = conv_transpose2d(scores, w, stride=2, pad=1)
25print(up.shape, up[2].round(3)) # (6, 6) [0. 0.25 0.75 0.875 0.625 0.375]
FCN's bilinear upsampling filter and a transposed convolution in numpy (runs as is)

Check yourself

A 320 × 320 image goes through convolutionalised VGG16 with no extra padding. What size is the score map before upsampling?

Check yourself

What does FCN-16s add to FCN-32s?

Remember

  • ✓fc layers = convolutions: fc6 → 7×77\times7 conv, fc7 → 1×11\times1 conv; any input size gives a score grid with stride 32.
  • ✓Scores are upsampled by transposed convolutions initialised to bilinear (kernel 2f, stride f).
  • ✓Skip fusion sums 1×1-conv scores from pool4 and pool3: z=U8(U2(U2(s32)+s16)+s8)z=U_8(U_2(U_2(s_{32})+s_{16})+s_8).
  • ✓Mean IU 59.4 → 62.4 → 62.7 for FCN-32s/16s/8s; 62.2 on VOC 2012 test; about 175 ms per image.
  • ✓Trained end to end with per-pixel cross-entropy on whole images; the template for every later segmentation network.
← back to Deep Learning roadmap