Perceptual loss

Compare images in the feature space of a pretrained network instead of pixel space. Feature distances track content, Gram matrices track texture and style, and LPIPS calibrates the distance to human judgments.

12 min read998 wordsadvanced0 viewsupdated 2026-10-11

Definition

Perceptual loss is any loss that measures the distance between an output y^\hat y and a target yy after passing both through a fixed, pretrained network ϕ\phi, usually VGG: ℓ=∑jλj d(ϕj(y^),ϕj(y))\ell=\sum_j \lambda_j\, d\big(\phi_j(\hat y),\phi_j(y)\big) over a few layers jj. Its two classic forms are the feature reconstruction loss, which compares activations position by position, and the style loss, which compares their Gram matrices (Gatys et al., 2015; Johnson, Alahi & Fei-Fei, 2016).

Pixel losses disagree with human eyes. Shift a photograph by one pixel and the squared error is large, though nobody can see the difference. Blur it slightly and the squared error is small, though everybody can. As the Reconstruction loss article shows, per-pixel losses are minimized by the mean of the plausible outputs, which is blurry. A perceptual loss moves the comparison into a representation where edges, textures and object parts are explicit and small misalignments matter less.

The idea came out of style transfer. Gatys, Ecker & Bethge (2015) synthesized images by gradient descent on the pixels, matching VGG-19 activations of a content photo and Gram matrices of a painting. Johnson, Alahi & Fei-Fei (2016) used the same two terms as training losses for a feed-forward network, giving real-time style transfer about three orders of magnitude faster than Gatys's optimization, and ×4 / ×8 super-resolution with sharper edges than per-pixel training. Dosovitskiy & Brox (2016) proposed similar feature-space losses at the same time. Zhang et al. (2018) asked how well such distances match human judgments, and fitted LPIPS to hundreds of thousands of human comparisons.

Feature reconstruction (content) loss

ℓfeatϕ,j(y^,y)=1CjHjWj ∥ϕj(y^)−ϕj(y)∥22\ell_{\text{feat}}^{\phi,j}(\hat y,y)=\frac{1}{C_jH_jW_j}\,\big\|\phi_j(\hat y)-\phi_j(y)\big\|_2^2
ϕj(⋅)\phi_j(\cdot)
activations of layer jj of the frozen network, a Cj×Hj×WjC_j\times H_j\times W_j tensor
jj
Johnson et al. use VGG-16 relu2_2 for super-resolution and relu3_3 for style transfer

Early layers respond to edges and colour blobs at almost every position, so matching them is close to matching pixels. Deeper layers have passed through several max-pools and respond to parts and textures over large receptive fields. Matching them preserves content and layout but tolerates small shifts and changes in fine texture. Johnson et al. found that reconstructing from higher layers keeps the image structure while letting colour and exact texture drift.

Style loss: Gram matrices

Reshape layer jj's activations into a Cj×HjWjC_j\times H_jW_j matrix ψj\psi_j, one row per channel. The Gram matrix records which channels fire together, averaged over all positions:

Gj(y)cc′=1CjHjWj∑h,wϕj(y)c,h,w ϕj(y)c′,h,w=ψjψj⊤CjHjWj,ℓstyleϕ,j=∥Gj(y^)−Gj(y)∥F2G_j(y)_{cc'}=\frac{1}{C_jH_jW_j}\sum_{h,w}\phi_j(y)_{c,h,w}\,\phi_j(y)_{c',h,w}=\frac{\psi_j\psi_j^{\top}}{C_jH_jW_j},\qquad \ell_{\text{style}}^{\phi,j}=\big\|G_j(\hat y)-G_j(y)\big\|_F^2
GjG_j
a Cj×CjC_j\times C_j matrix of uncentred feature correlations; spatial positions are summed out
∥⋅∥F\|\cdot\|_F
Frobenius norm. Gatys et al. use slightly different normalizing constants; only the per-layer weights change

Summing over positions throws away where things are and keeps what co-occurs: stroke orientation, colour palettes, repeated motifs. Any rearrangement of the positions leaves GG unchanged, so a circularly shifted texture has style loss exactly 0. Style losses are summed over several layers (Johnson et al. use relu1_2, relu2_2, relu3_3 and relu4_3), so both fine and coarse texture statistics are matched. Since ϕ\phi is fixed and differentiable, the gradient reaches the image by ordinary backpropagation through the frozen network:

∂ℓstyle∂ψj(y^)=4CjHjWj(Gj(y^)−Gj(y)) ψj(y^),∂ℓfeat∂ϕj(y^)=2CjHjWj(ϕj(y^)−ϕj(y))\frac{\partial\ell_{\text{style}}}{\partial\psi_j(\hat y)}=\frac{4}{C_jH_jW_j}\big(G_j(\hat y)-G_j(y)\big)\,\psi_j(\hat y),\qquad \frac{\partial\ell_{\text{feat}}}{\partial\phi_j(\hat y)}=\frac{2}{C_jH_jW_j}\big(\phi_j(\hat y)-\phi_j(y)\big)
44
22 from the square, 22 because GG is symmetric and ψj\psi_j appears twice in ψjψj⊤\psi_j\psi_j^\top

At a glance

Feature network
frozen, ImageNet-pretrained VGG-16/19 is standard; LPIPS also offers AlexNet and SqueezeNet
Content term
feature MSE at one mid-level layer; tolerant to small shifts, sensitive to blur
Style term
Gram-matrix distance summed over several layers; position-free
Usually combined with
a pixel L1/L2 term, an adversarial term (SRGAN, image-to-image), and sometimes total-variation regularization
Cost
one extra forward and backward pass through the feature network per training step
Inputs
must be preprocessed like the network's training data, e.g. ImageNet mean/std normalization for torchvision VGG; LPIPS expects [−1,1][-1,1]
target y (64 px, stripes) and distorted x: shift 1 px · blur σ 0.0 · noise 0.0000.51━ y ━ x · dotted lines = 8-px pooling windowsφ(y)φ(x)filters → ReLU → avg-pool 8 · 5 channels × 8 windowsedge +edge −bar +bar −brightG(x) − G(y), Gram 5×5scale ±0.020 · position-freecandidate · pixel MSE · feature · style (Gram) · rank 1 = judged closestshift 2 px0.1465 #43.3e-4 #20 #1shift 4 px0.2622 #58.9e-4 #30 #1blur σ=1.50.0268 #21.6e-3 #46.0e-3 #4noise 0.150.0259 #12.4e-3 #59.0e-3 #5your x0.0733 #32.8e-4 #10 #1‖∂ℓ/∂x‖ for your x: pixel 0.0677 · feature 1.2e-3 · style 0
fig.A 1-D stand-in for VGG: 64-pixel stripes passed through five fixed filters (two edge polarities, two bar polarities, brightness), ReLU and 8-pixel average pooling. The table ranks four distortions by each loss. Pixel MSE calls noise (0.0259) and a σ = 1.5 blur (0.0268) closest and a 2-pixel shift 5.5× worse (0.1465). The pooled feature loss reverses the order: the shift costs 3.3e-4, the blur 1.6e-3 (4.9× more), because pooling windows absorb small misalignments while blur weakens every edge response. The Gram style loss is exactly 0 for any integer shift. Drag the sliders to combine distortions; the bottom line shows each loss's gradient norm with respect to the image.

LPIPS: a learned perceptual metric

Raw VGG distances are already better aligned with people than PSNR or SSIM, but different channels have very different scales. LPIPS (Learned Perceptual Image Patch Similarity; Zhang, Isola, Efros, Shechtman & Wang, 2018) normalizes each feature vector to unit length along the channel axis, weights the channels, and averages over space and layers:

dLPIPS(x,x0)=∑l1HlWl∑h,w∥ wl⊙(ϕ^l(x)hw−ϕ^l(x0)hw)∥22,ϕ^l(x)hw=ϕl(x)hw∥ϕl(x)hw∥2d_{\text{LPIPS}}(x,x_0)=\sum_l\frac{1}{H_lW_l}\sum_{h,w}\big\|\,w_l\odot\big(\hat\phi_l(x)_{hw}-\hat\phi_l(x_0)_{hw}\big)\big\|_2^2,\qquad \hat\phi_l(x)_{hw}=\frac{\phi_l(x)_{hw}}{\|\phi_l(x)_{hw}\|_2}
wl∈RClw_l\in\mathbb R^{C_l}
per-channel weights fitted to human two-alternative judgments (the BAPPS dataset); the backbone stays frozen in the default "lin" version
ϕ^l\hat\phi_l
unit-normalized feature vector at one spatial position

The paper's headline finding is that deep features from very different networks, supervised, self-supervised or unsupervised, all agree with humans far better than classic metrics. That is the "unreasonable effectiveness" of the title. LPIPS is now a standard evaluation metric for super-resolution, image synthesis and novel-view rendering, and it is also used as a training loss. Lower is more similar, and 0 means identical features.

Worked example: a tiny Gram matrix

Two channels at three positions, ψ(y^)=(102011)\psi(\hat y)=\begin{pmatrix}1&0&2\\0&1&1\end{pmatrix}. The target is the same map shifted one position to the right (circularly), ψ(y)=(210101)\psi(y)=\begin{pmatrix}2&1&0\\1&0&1\end{pmatrix}. Normalize by the 3 positions, as the figure does (verified in Node):

  1. 1G(y^)=13(1+0+40+0+220+1+1)=(1.6670.6670.6670.667)G(\hat y)=\tfrac13\begin{pmatrix}1+0+4 & 0+0+2\\ 2 & 0+1+1\end{pmatrix}=\begin{pmatrix}1.667&0.667\\0.667&0.667\end{pmatrix}. G(y)G(y) is identical, because each entry is a sum over positions and the shift only reorders them.
  2. 2Style loss ∥G(y^)−G(y)∥F2=0\|G(\hat y)-G(y)\|_F^2=0, and its gradient is 0 too.
  3. 3Feature loss: ψ(y^)−ψ(y)=(−1−12−110)\psi(\hat y)-\psi(y)=\begin{pmatrix}-1&-1&2\\-1&1&0\end{pmatrix}, so the mean square is (1+1+4+1+1+0)/6=1.333(1+1+4+1+1+0)/6=1.333.
  4. 4Now flatten each channel to its mean instead, (1110.6670.6670.667)\begin{pmatrix}1&1&1\\0.667&0.667&0.667\end{pmatrix}, a crude "blur". The feature loss falls to 0.4440.444, but the Gram matrix becomes (10.6670.6670.444)\begin{pmatrix}1&0.667\\0.667&0.444\end{pmatrix} and the style loss rises from 0 to 0.6672+0.2222=0.4940.667^2+0.222^2=0.494. Without pooling, the feature loss penalizes the shift more than the blur, while the style loss penalizes only the blur.
python
1import torch, torch.nn as nn, torch.nn.functional as F
2from torchvision.models import vgg16, VGG16_Weights
3
4class VGGFeatures(nn.Module):
5 # indices in vgg16().features: relu1_2=3, relu2_2=8, relu3_3=15, relu4_3=22
6 def __init__(self, layers=(3, 8, 15, 22)):
7 super().__init__()
8 net = vgg16(weights=VGG16_Weights.IMAGENET1K_V1).features[: max(layers) + 1].eval()
9 for p in net.parameters():
10 p.requires_grad_(False) # frozen: gradients flow to the image only
11 self.net, self.layers = net, set(layers)
12 self.register_buffer("mean", torch.tensor([0.485, 0.456, 0.406]).view(1, 3, 1, 1))
13 self.register_buffer("std", torch.tensor([0.229, 0.224, 0.225]).view(1, 3, 1, 1))
14
15 def forward(self, x): # x in [0, 1], shape (B, 3, H, W)
16 x, feats = (x - self.mean) / self.std, []
17 for i, layer in enumerate(self.net):
18 x = layer(x)
19 if i in self.layers:
20 feats.append(x)
21 return feats
22
23def gram(f): # (B, C, H, W) -> (B, C, C)
24 B, C, H, W = f.shape
25 f = f.flatten(2)
26 return f @ f.transpose(1, 2) / (C * H * W)
27
28vgg = VGGFeatures()
29def perceptual_loss(y_hat, y, style_target, w_feat=1.0, w_style=1e5):
30 fy_hat, fy, fs = vgg(y_hat), vgg(y), vgg(style_target)
31 feat = F.mse_loss(fy_hat[1], fy[1]) # relu2_2 content term
32 style = sum(F.mse_loss(gram(a), gram(b), reduction="sum") / a.shape[0]
33 for a, b in zip(fy_hat, fs))
34 return w_feat * feat + w_style * style
35
36# LPIPS: pip install lpips; inputs scaled to [-1, 1]
37import lpips
38d = lpips.LPIPS(net="alex") # or "vgg"
39dist = d(y_hat * 2 - 1, y * 2 - 1) # (B, 1, 1, 1); differentiable
VGG-16 feature and style losses (Johnson et al. layers) and LPIPS

Use when

  • +Better aligned with human similarity judgments than pixel MSE, PSNR or SSIM
  • +Tolerates small misalignments, so it does not force blurry averages
  • +Gram matrices give a position-free texture statistic for style and texture synthesis
  • +Training-free in its basic form: any frozen pretrained network works, and LPIPS is a drop-in package

Avoid when

  • −Extra forward and backward pass through a large network on every step
  • −Inherits the feature network's biases, input range and domain (ImageNet photos), so it is weaker on medical, satellite or line-art images
  • −Can be gamed: outputs with artefacts that match features but look wrong
  • −Many weights to tune (layers, feature vs style, pixel and adversarial terms)

Check yourself

In the figure, the striped signal is circularly shifted by 3 pixels. Which loss is exactly zero?

Check yourself

In the figure, why does the pooled feature loss rank a 2-pixel shift as closer than a σ = 1.5 blur, when pixel MSE says the opposite?

Remember

  • ✓Perceptual loss = distance between frozen-network features of output and target, 1CHW∥ϕj(y^)−ϕj(y)∥2\frac{1}{CHW}\|\phi_j(\hat y)-\phi_j(y)\|^2.
  • ✓Gram matrices G=ψψ⊤/CHWG=\psi\psi^\top/CHW discard position and keep channel co-occurrence: a style/texture loss.
  • ✓Deeper layers tolerate small shifts and punish blur, which is why perceptual training gives sharper super-resolution than MSE.
  • ✓LPIPS unit-normalizes features per position, applies learned channel weights, and is calibrated to human judgments.
  • ✓Preprocess inputs as the feature network expects, freeze it, and pair the loss with pixel and often adversarial terms.
← back to Deep Learning roadmap