# Perceptual loss

> Compare images in the feature space of a pretrained network instead of pixel space. Feature distances track content, Gram matrices track texture and style, and LPIPS calibrates the distance to human judgments.

Source: https://www.mlfoundry.in/learn/perceptual-loss  
Section: Deep Learning › Loss Functions  
Level: advanced · 12 min read · updated 2026-10-11

**Perceptual loss:** is any loss that measures the distance between an output $\hat y$ and a target $y$ after passing both through a fixed, pretrained network $\phi$, usually VGG: $\ell=\sum_j \lambda_j\, d\big(\phi_j(\hat y),\phi_j(y)\big)$ over a few layers $j$. Its two classic forms are the **feature reconstruction loss**, which compares activations position by position, and the **style loss**, which compares their Gram matrices (Gatys et al., 2015; Johnson, Alahi & Fei-Fei, 2016).

Pixel losses disagree with human eyes. Shift a photograph by one pixel and the squared error is large, though nobody can see the difference. Blur it slightly and the squared error is small, though everybody can. As the **Reconstruction loss** article shows, per-pixel losses are minimized by the mean of the plausible outputs, which is blurry. A perceptual loss moves the comparison into a representation where edges, textures and object parts are explicit and small misalignments matter less.

The idea came out of style transfer. Gatys, Ecker & Bethge (2015) synthesized images by gradient descent on the pixels, matching VGG-19 activations of a content photo and Gram matrices of a painting. Johnson, Alahi & Fei-Fei (2016) used the same two terms as **training losses** for a feed-forward network, giving real-time style transfer about three orders of magnitude faster than Gatys's optimization, and ×4 / ×8 super-resolution with sharper edges than per-pixel training. Dosovitskiy & Brox (2016) proposed similar feature-space losses at the same time. Zhang et al. (2018) asked how well such distances match human judgments, and fitted **LPIPS** to hundreds of thousands of human comparisons.

## Feature reconstruction (content) loss

$$
\ell_{\text{feat}}^{\phi,j}(\hat y,y)=\frac{1}{C_jH_jW_j}\,\big\|\phi_j(\hat y)-\phi_j(y)\big\|_2^2
$$

- $\phi_j(\cdot)$: activations of layer $j$ of the frozen network, a $C_j\times H_j\times W_j$ tensor
- $j$: Johnson et al. use VGG-16 `relu2_2` for super-resolution and `relu3_3` for style transfer

Early layers respond to edges and colour blobs at almost every position, so matching them is close to matching pixels. Deeper layers have passed through several max-pools and respond to parts and textures over large receptive fields. Matching them preserves content and layout but tolerates small shifts and changes in fine texture. Johnson et al. found that reconstructing from higher layers keeps the image structure while letting colour and exact texture drift.

## Style loss: Gram matrices

Reshape layer $j$'s activations into a $C_j\times H_jW_j$ matrix $\psi_j$, one row per channel. The Gram matrix records which channels fire **together**, averaged over all positions:

$$
G_j(y)_{cc'}=\frac{1}{C_jH_jW_j}\sum_{h,w}\phi_j(y)_{c,h,w}\,\phi_j(y)_{c',h,w}=\frac{\psi_j\psi_j^{\top}}{C_jH_jW_j},\qquad \ell_{\text{style}}^{\phi,j}=\big\|G_j(\hat y)-G_j(y)\big\|_F^2
$$

- $G_j$: a $C_j\times C_j$ matrix of uncentred feature correlations; spatial positions are summed out
- $\|\cdot\|_F$: Frobenius norm. Gatys et al. use slightly different normalizing constants; only the per-layer weights change

Summing over positions throws away **where** things are and keeps **what** co-occurs: stroke orientation, colour palettes, repeated motifs. Any rearrangement of the positions leaves $G$ unchanged, so a circularly shifted texture has style loss exactly 0. Style losses are summed over several layers (Johnson et al. use `relu1_2`, `relu2_2`, `relu3_3` and `relu4_3`), so both fine and coarse texture statistics are matched. Since $\phi$ is fixed and differentiable, the gradient reaches the image by ordinary backpropagation through the frozen network:

$$
\frac{\partial\ell_{\text{style}}}{\partial\psi_j(\hat y)}=\frac{4}{C_jH_jW_j}\big(G_j(\hat y)-G_j(y)\big)\,\psi_j(\hat y),\qquad \frac{\partial\ell_{\text{feat}}}{\partial\phi_j(\hat y)}=\frac{2}{C_jH_jW_j}\big(\phi_j(\hat y)-\phi_j(y)\big)
$$

- $4$: $2$ from the square, $2$ because $G$ is symmetric and $\psi_j$ appears twice in $\psi_j\psi_j^\top$

**At a glance**

- **Feature network:** frozen, ImageNet-pretrained VGG-16/19 is standard; LPIPS also offers AlexNet and SqueezeNet
- **Content term:** feature MSE at one mid-level layer; tolerant to small shifts, sensitive to blur
- **Style term:** Gram-matrix distance summed over several layers; position-free
- **Usually combined with:** a pixel L1/L2 term, an adversarial term (SRGAN, image-to-image), and sometimes total-variation regularization
- **Cost:** one extra forward and backward pass through the feature network per training step
- **Inputs:** must be preprocessed like the network's training data, e.g. ImageNet mean/std normalization for torchvision VGG; LPIPS expects $[-1,1]$

> Figure: A 1-D stand-in for VGG: 64-pixel stripes passed through five fixed filters (two edge polarities, two bar polarities, brightness), ReLU and 8-pixel average pooling. The table ranks four distortions by each loss. Pixel MSE calls noise (0.0259) and a σ = 1.5 blur (0.0268) closest and a 2-pixel shift 5.5× worse (0.1465). The pooled feature loss reverses the order: the shift costs 3.3e-4, the blur 1.6e-3 (4.9× more), because pooling windows absorb small misalignments while blur weakens every edge response. The Gram style loss is exactly 0 for any integer shift. Drag the sliders to combine distortions; the bottom line shows each loss's gradient norm with respect to the image.

## LPIPS: a learned perceptual metric

Raw VGG distances are already better aligned with people than PSNR or SSIM, but different channels have very different scales. **LPIPS** (Learned Perceptual Image Patch Similarity; Zhang, Isola, Efros, Shechtman & Wang, 2018) normalizes each feature vector to unit length along the channel axis, weights the channels, and averages over space and layers:

$$
d_{\text{LPIPS}}(x,x_0)=\sum_l\frac{1}{H_lW_l}\sum_{h,w}\big\|\,w_l\odot\big(\hat\phi_l(x)_{hw}-\hat\phi_l(x_0)_{hw}\big)\big\|_2^2,\qquad \hat\phi_l(x)_{hw}=\frac{\phi_l(x)_{hw}}{\|\phi_l(x)_{hw}\|_2}
$$

- $w_l\in\mathbb R^{C_l}$: per-channel weights fitted to human two-alternative judgments (the BAPPS dataset); the backbone stays frozen in the default "lin" version
- $\hat\phi_l$: unit-normalized feature vector at one spatial position

The paper's headline finding is that deep features from very different networks, supervised, self-supervised or unsupervised, all agree with humans far better than classic metrics. That is the "unreasonable effectiveness" of the title. LPIPS is now a standard **evaluation** metric for super-resolution, image synthesis and novel-view rendering, and it is also used as a training loss. Lower is more similar, and 0 means identical features.

## Worked example: a tiny Gram matrix

Two channels at three positions, $\psi(\hat y)=\begin{pmatrix}1&0&2\\0&1&1\end{pmatrix}$. The target is the same map shifted one position to the right (circularly), $\psi(y)=\begin{pmatrix}2&1&0\\1&0&1\end{pmatrix}$. Normalize by the 3 positions, as the figure does (verified in Node):

1. $G(\hat y)=\tfrac13\begin{pmatrix}1+0+4 & 0+0+2\\ 2 & 0+1+1\end{pmatrix}=\begin{pmatrix}1.667&0.667\\0.667&0.667\end{pmatrix}$. $G(y)$ is identical, because each entry is a sum over positions and the shift only reorders them.
2. Style loss $\|G(\hat y)-G(y)\|_F^2=0$, and its gradient is 0 too.
3. Feature loss: $\psi(\hat y)-\psi(y)=\begin{pmatrix}-1&-1&2\\-1&1&0\end{pmatrix}$, so the mean square is $(1+1+4+1+1+0)/6=1.333$.
4. Now flatten each channel to its mean instead, $\begin{pmatrix}1&1&1\\0.667&0.667&0.667\end{pmatrix}$, a crude "blur". The feature loss falls to $0.444$, but the Gram matrix becomes $\begin{pmatrix}1&0.667\\0.667&0.444\end{pmatrix}$ and the style loss rises from 0 to $0.667^2+0.222^2=0.494$. Without pooling, the feature loss penalizes the shift more than the blur, while the style loss penalizes only the blur.

```python
import torch, torch.nn as nn, torch.nn.functional as F
from torchvision.models import vgg16, VGG16_Weights

class VGGFeatures(nn.Module):
    # indices in vgg16().features: relu1_2=3, relu2_2=8, relu3_3=15, relu4_3=22
    def __init__(self, layers=(3, 8, 15, 22)):
        super().__init__()
        net = vgg16(weights=VGG16_Weights.IMAGENET1K_V1).features[: max(layers) + 1].eval()
        for p in net.parameters():
            p.requires_grad_(False)                    # frozen: gradients flow to the image only
        self.net, self.layers = net, set(layers)
        self.register_buffer("mean", torch.tensor([0.485, 0.456, 0.406]).view(1, 3, 1, 1))
        self.register_buffer("std", torch.tensor([0.229, 0.224, 0.225]).view(1, 3, 1, 1))

    def forward(self, x):                              # x in [0, 1], shape (B, 3, H, W)
        x, feats = (x - self.mean) / self.std, []
        for i, layer in enumerate(self.net):
            x = layer(x)
            if i in self.layers:
                feats.append(x)
        return feats

def gram(f):                                           # (B, C, H, W) -> (B, C, C)
    B, C, H, W = f.shape
    f = f.flatten(2)
    return f @ f.transpose(1, 2) / (C * H * W)

vgg = VGGFeatures()
def perceptual_loss(y_hat, y, style_target, w_feat=1.0, w_style=1e5):
    fy_hat, fy, fs = vgg(y_hat), vgg(y), vgg(style_target)
    feat = F.mse_loss(fy_hat[1], fy[1])                # relu2_2 content term
    style = sum(F.mse_loss(gram(a), gram(b), reduction="sum") / a.shape[0]
                for a, b in zip(fy_hat, fs))
    return w_feat * feat + w_style * style

# LPIPS: pip install lpips; inputs scaled to [-1, 1]
import lpips
d = lpips.LPIPS(net="alex")                            # or "vgg"
dist = d(y_hat * 2 - 1, y * 2 - 1)                     # (B, 1, 1, 1); differentiable
```

VGG-16 feature and style losses (Johnson et al. layers) and LPIPS

> **Wrong preprocessing, silent failure:** The feature network only produces meaningful features for inputs that look like its training data. Feeding $[0,1]$ images to a torchvision VGG without ImageNet mean/std normalization, or $[0,1]$ images to LPIPS (which expects $[-1,1]$), still runs and still gives a loss, just a worse-aligned one. Keep the feature network in `eval()` mode and its parameters frozen. Remember also that `F.mse_loss` averages, so the feature and style weights depend on the reduction you use: style weights of $10^3$ to $10^6$ are common.

> **Why it sharpens, and its artefacts:** A pixel loss charges a correct but slightly misplaced texture almost as much as a wrong one, so its safest answer is a smooth average. In feature space, a plausible texture at a slightly wrong phase scores almost as well as the exact one, while a smooth average lights up none of the edge and texture channels and scores badly. That rewards committing to sharp detail. The price is that the network can satisfy the features with patterns people can see: checkerboard and high-frequency artefacts that excite the right VGG channels. This is why perceptual terms are usually paired with a pixel term and often an adversarial one.

> **Where you meet it:** Fast style transfer (Johnson et al.), SRGAN and ESRGAN super-resolution (VGG feature loss plus an adversarial loss), image-to-image translation and inpainting, VQGAN and the autoencoders of latent diffusion models (LPIPS plus a patch adversarial loss, so the latent decoder reconstructs sharp images), and as an evaluation metric almost everywhere images are generated.

**Strengths**

- Better aligned with human similarity judgments than pixel MSE, PSNR or SSIM
- Tolerates small misalignments, so it does not force blurry averages
- Gram matrices give a position-free texture statistic for style and texture synthesis
- Training-free in its basic form: any frozen pretrained network works, and LPIPS is a drop-in package

**Limitations**

- Extra forward and backward pass through a large network on every step
- Inherits the feature network's biases, input range and domain (ImageNet photos), so it is weaker on medical, satellite or line-art images
- Can be gamed: outputs with artefacts that match features but look wrong
- Many weights to tune (layers, feature vs style, pixel and adversarial terms)

**Question:** In the figure, the striped signal is circularly shifted by 3 pixels. Which loss is exactly zero?

- A. Pixel MSE
- B. Feature reconstruction loss at an early layer
- C. Gram-matrix style loss
- D. LPIPS

**Answer:** C. Gram-matrix style loss — The Gram matrix sums products of channel activations over all positions. The figure's filters wrap around, so a circular shift of the input only reorders the positions of every feature map, and $G(\hat y)=G(y)$ exactly. (With the zero padding of a real CNN, borders make it only approximately zero.) The pixel and feature losses compare activations position by position, so they change. LPIPS also compares features position by position before averaging, so it is small but not zero.

**Question:** In the figure, why does the pooled feature loss rank a 2-pixel shift as closer than a σ = 1.5 blur, when pixel MSE says the opposite?

- A. The feature network is randomly initialized
- B. Average pooling over 8-pixel windows absorbs small shifts, while blur reduces every edge and bar response
- C. The feature loss uses L1 instead of L2
- D. The blur changes the image's mean brightness

**Answer:** B. Average pooling over 8-pixel windows absorbs small shifts, while blur reduces every edge and bar response — Each pooled feature sums the ReLU edge responses over a window that covers one full stripe period, so moving the stripes by 2 pixels barely changes it (3.3e-4). Blurring lowers the edge and bar responses everywhere (1.6e-3). Pixel MSE has no pooling, so it is dominated by the misaligned edges of the shifted signal.

**Key points**

- Perceptual loss = distance between frozen-network features of output and target, $\frac{1}{CHW}\|\phi_j(\hat y)-\phi_j(y)\|^2$.
- Gram matrices $G=\psi\psi^\top/CHW$ discard position and keep channel co-occurrence: a style/texture loss.
- Deeper layers tolerate small shifts and punish blur, which is why perceptual training gives sharper super-resolution than MSE.
- LPIPS unit-normalizes features per position, applies learned channel weights, and is calibrated to human judgments.
- Preprocess inputs as the feature network expects, freeze it, and pair the loss with pixel and often adversarial terms.

---

From MLFoundry (https://www.mlfoundry.in). The web page adds interactive figures and diagrams.
