# FCN

> FCN (Long, Shelhamer, Darrell, 2015) turns a classifier's fc layers into convolutions, predicts class scores at stride 32, upsamples them with transposed convolutions and adds pool4 and pool3 scores (FCN-8s): 62.2 mIoU on VOC 2012.

Source: https://www.mlfoundry.in/learn/fcn  
Section: Deep Learning › Deep Learning for Computer Vision  
Level: core · 7 min read · updated 2026-10-15

**Fully convolutional network (FCN):** is a network made only of convolution, pooling and upsampling layers, so it maps an image of any size to a score map of matching size. Long, Shelhamer and Darrell ("Fully Convolutional Networks for Semantic Segmentation", CVPR 2015) adapted ImageNet classifiers this way and trained them end to end for pixelwise **Image segmentation**, improving PASCAL VOC mean IU by about 20% relative to the previous best.

Before FCN, segmentation systems classified region proposals or superpixels with a CNN and merged the results (R-CNN-based SDS, Hariharan et al., 2014), or ran a patch classifier at every pixel. Both repeat almost the same computation for overlapping inputs, and both need pipelines of separate stages. FCN produces all pixel labels in one forward pass, about 175 ms per image against roughly 50 s for SDS.

## Convolutionalising a classifier

A fully connected layer over a $7\times7\times512$ input is exactly a $7\times7$ convolution with as many filters as outputs. So VGG16's fc6 becomes a $7\times7$ conv with 4096 filters and fc7 a $1\times1$ conv with 4096 filters, with the same weights. On a larger image the network no longer outputs one vector but a grid of them, one per $32$-pixel step: it is the classifier slid over the image, with the shared convolutions computed once.

$$
h_{\text{out}}=\Big\lfloor\frac{H}{32}\Big\rfloor-7+1,\qquad s_{k}(i,j)=\sum_{c}w_{k,c}\,f_c(i,j)+b_k
$$

- $h_{\text{out}}$: side of the fc6 output map for an $H\times H$ input (pool5 has stride 32, fc6 is a valid $7\times7$ conv)
- $s_k(i,j)$: score of class $k$ at cell $(i,j)$: a $1\times1$ convolution of the features $f$, 21 outputs for PASCAL VOC (20 classes + background)

The released Caffe models pad the first convolution by 100 pixels and crop the output back to the input size, so small images still give a valid fc6 map. A $1\times1$ "score" convolution then maps the 4096 channels to 21 class scores.

> Figure: FCN-VGG16. The convolutionalised classifier gives 21 class scores at stride 32. FCN-32s upsamples them ×32 in one step. FCN-16s first upsamples them ×2 with a learnable transposed convolution and adds 1×1-conv scores computed from pool4 (stride 16); FCN-8s repeats that with pool3 (stride 8), then upsamples ×8. Mean IU on a PASCAL VOC 2011 validation subset rises from 59.4 to 62.4 to 62.7.

## Upsampling and skip fusion

The stride-32 scores are coarse. FCN upsamples them with a **Transposed convolution** of stride $f$ and kernel size $2f$ ($64$ for $f=32$, $4$ for $f=2$), initialised to bilinear interpolation. The final upsampler is kept fixed at bilinear; the intermediate $2\times$ ones are learned. Each kernel row is a tent:

$$
w(t)=1-\frac{|t-c|}{f},\quad c=f-\tfrac12,\ t=0,\dots,2f-1;\qquad z=U_{8}\Big(U_{2}\big(U_{2}(s_{32})+s_{16}\big)+s_{8}\Big)
$$

- $w(t)$: 1D bilinear kernel for even size $2f$; the 2D kernel is $w(t)\,w(t')$
- $s_{32},\,s_{16},\,s_{8}$: 21-channel scores from conv7, pool4 and pool3 (the last two via new $1\times1$ convs)
- $U_f$: upsampling by $f$ (transposed conv); $z$ is the FCN-8s output at input resolution

Deep layers know *what* is there but see it at stride 32. Shallow layers have finer resolution but less semantics. Summing their scores, coarse to fine, gives the "deep jet" of the paper. The networks are trained in stages, FCN-32s then 16s then 8s, each initialised from the previous one. Going below stride 8 gave diminishing returns.

**Configuration and results (from the paper)**

- **Backbone:** VGG16 (ImageNet), fc6 → 7×7 conv, 4096 filters; fc7 → 1×1 conv, 4096; score → 1×1 conv, 21
- **Upsamplers:** 2× steps: kernel 4, stride 2 (learned); final: kernel 64 / 32 / 16 at stride 32 / 16 / 8 (bilinear, fixed)
- **Loss:** per-pixel multinomial logistic loss over the whole image (no patch sampling)
- **VOC 2011 val subset:** mean IU 59.4 (FCN-32s), 62.4 (FCN-16s), 62.7 (FCN-8s)
- **Test sets:** FCN-8s 62.7 mean IU on VOC 2011 test and 62.2 on VOC 2012 test, a 20% relative gain over the previous best (SDS, 52.6 on 2011)

> Figure: An oracle FCN: each coarse cell holds the true object fraction, the best a stride-s score map can do. Default FCN-32s (4×4 cells, bilinear ×32): object IoU 0.539, mean IoU 0.720, and the 5-pixel pole is invisible because its cells score at most 0.16. FCN-16s: object IoU 0.694. FCN-8s: object IoU 0.904, mean IoU 0.941, and 70.4% of the pole pixels are found. Switch FCN-8s to nearest upsampling and object IoU falls to 0.773: blocky edges add 441 false-positive pixels.

The real gains were smaller (59.4 to 62.7) because learned scores are not oracle scores, and thin structures make up a small part of VOC pixels. The same resolution problem led to **Dilated convolution** in **DeepLab** and to the full decoders of **U-Net**.

## Worked example: output size and a 2× upsample

VGG16 made fully convolutional, without the 100-pixel padding: pool5 = H / 32, fc6 output = pool5 − 6

| Input H × H | pool5 | fc6 / score map | Meaning |
| --- | --- | --- | --- |
| 224 × 224 | 7 × 7 | 1 × 1 | the original classifier: one prediction |
| 288 × 288 | 9 × 9 | 3 × 3 | classifier at 9 window positions, step 32 px |
| 384 × 384 | 12 × 12 | 6 × 6 | 36 windows, conv features computed once |
| 512 × 512 | 16 × 16 | 10 × 10 | 100 windows |

**Upsample the score row (0, 1, 0.5) by 2 with the 4-tap bilinear kernel (0.25, 0.75, 0.75, 0.25), stride 2, padding 1**

1. Each input stamps a scaled kernel at offset 2i: 0 × kernel at 0–3, 1 × kernel at 2–5, 0.5 × kernel = (0.125, 0.375, 0.375, 0.125) at 4–7.
2. Sum the stamps: (0, 0, 0.25, 0.75, 0.875, 0.625, 0.375, 0.125).
3. Crop one value at each end (padding 1): (0, 0.25, 0.75, 0.875, 0.625, 0.375), twice the input length.
4. This equals bilinear interpolation (align_corners = False) except at the last position: 0.375 instead of 0.5, because the transposed conv treats the outside as zero. Training then adjusts the 2× kernels freely.

```python
import numpy as np

def bilinear_kernel(size):
    """FCN's upsampling filter: kernel size 2f - f % 2 for factor f (4 for 2×, 64 for 32×)."""
    f = (size + 1) // 2
    c = f - 1 if size % 2 else f - 0.5
    k = 1 - np.abs(np.arange(size) - c) / f
    return np.outer(k, k)

def conv_transpose2d(x, w, stride, pad):
    """Single-channel transposed convolution: every input pixel stamps a scaled copy of w."""
    n, k = x.shape[0], w.shape[0]
    y = np.zeros(((n - 1) * stride + k,) * 2)
    for i in range(n):
        for j in range(n):
            y[i * stride:i * stride + k, j * stride:j * stride + k] += x[i, j] * w
    return y[pad:-pad, pad:-pad] if pad else y

w = bilinear_kernel(4)
print(w[1])                                   # [0.1875 0.5625 0.5625 0.1875] = 0.75 * [.25 .75 .75 .25]
print(bilinear_kernel(64).shape)              # (64, 64): the FCN-32s upsampler, stride 32

scores = np.array([[0.0, 1.0, 0.5]] * 3)      # a 3×3 class-score map, constant down the columns
up = conv_transpose2d(scores, w, stride=2, pad=1)
print(up.shape, up[2].round(3))               # (6, 6) [0.    0.25  0.75  0.875 0.625 0.375]
```

FCN's bilinear upsampling filter and a transposed convolution in numpy (runs as is)

> **torchvision's FCN is not FCN-8s:** `torchvision.models.segmentation.fcn_resnet50` keeps the name but uses a dilated ResNet backbone at output stride 8, a $3\times3$ conv head and one bilinear upsample, with no skip fusion. It is closer in design to **DeepLab** without ASPP. For the paper's model, use the original Caffe definitions.

> **Why the skips are summed:** Scores live in the same 21-class space at every stride, so adding them is a residual correction: the pool4 term only has to fix the upsampled conv7 scores near boundaries. **U-Net** instead concatenates features and lets later convolutions decide how to mix them.

**Question:** A 320 × 320 image goes through convolutionalised VGG16 with no extra padding. What size is the score map before upsampling?

- A. 10 × 10
- B. 4 × 4
- C. 5 × 5
- D. 1 × 1

**Answer:** B. 4 × 4 — pool5 is $320/32=10$, and the valid $7\times7$ fc6 conv leaves $10-7+1=4$. The 1×1 layers keep $4\times4$.

**Question:** What does FCN-16s add to FCN-32s?

- A. A second backbone at half resolution
- B. Class scores from pool4 (1×1 conv), summed with the 2×-upsampled conv7 scores, then ×16 upsampling
- C. Concatenated pool4 features followed by two 3×3 convs
- D. Dilated convolutions in conv5

**Answer:** B. Class scores from pool4 (1×1 conv), summed with the 2×-upsampled conv7 scores, then ×16 upsampling — The skip is a $1\times1$ score layer on pool4 (stride 16). Its 21 scores are added to the conv7 scores after a learned $2\times$ transposed conv, and the sum is upsampled $\times16$. Mean IU went from 59.4 to 62.4.

**Key points**

- fc layers = convolutions: fc6 → $7\times7$ conv, fc7 → $1\times1$ conv; any input size gives a score grid with stride 32.
- Scores are upsampled by transposed convolutions initialised to bilinear (kernel 2f, stride f).
- Skip fusion sums 1×1-conv scores from pool4 and pool3: $z=U_8(U_2(U_2(s_{32})+s_{16})+s_8)$.
- Mean IU 59.4 → 62.4 → 62.7 for FCN-32s/16s/8s; 62.2 on VOC 2012 test; about 175 ms per image.
- Trained end to end with per-pixel cross-entropy on whole images; the template for every later segmentation network.

---

From MLFoundry (https://www.mlfoundry.in). The web page adds interactive figures and diagrams.
