# Swin Transformer

> A hierarchical vision Transformer: self-attention inside 7×7 windows (cost linear in image size), windows shifted every other block so information crosses their borders, and patch merging that halves resolution like a CNN stage. Swin-T: 28M params, 4.5 GFLOPs, 81.3%.

Source: https://www.mlfoundry.in/learn/swin-transformer  
Section: Deep Learning › Attention and Transformer Architectures  
Level: advanced · 14 min read · updated 2026-10-11

**Swin Transformer:** (**S**hifted **win**dow Transformer) is a vision Transformer that computes self-attention only within non-overlapping $M\times M$ windows of tokens ($M=7$), alternates between a regular and a **shifted** window layout in consecutive blocks so that neighbouring windows exchange information, and builds a four-stage **hierarchy** by merging $2\times2$ neighbouring tokens between stages. Its feature maps have strides 4, 8, 16, 32, exactly like a ResNet's.

Liu et al. introduced Swin in "Swin Transformer: Hierarchical Vision Transformer using Shifted Windows" (ICCV 2021, Marr Prize). They targeted two problems of **Vision Transformers (ViT)** as general backbones. First, global attention costs $O(N^2)$ in the number of tokens, which is prohibitive for detection and segmentation at high resolution. Second, ViT keeps one coarse scale ($16\times$ downsampled) throughout, while dense prediction wants a feature pyramid. Swin reintroduces two convolutional priors, **locality** and **hierarchy**, while keeping attention as the mixing operator.

It became the default Transformer backbone for dense prediction: Swin-L reached 58.7 box AP and 51.1 mask AP on COCO test-dev and 53.5 mIoU on ADE20K, state of the art at publication. On ImageNet-1K classification, Swin-T reached 81.3% top-1 at 4.5 GFLOPs, against 79.8% for DeiT-S at a similar cost (paper Table 1).

## Window attention and its cost

$$
\Omega(\mathrm{MSA})=4hwC^2+2(hw)^2C,\qquad \Omega(\mathrm{W\text{-}MSA})=4hwC^2+2M^2\,hw\,C
$$

- $h\times w$: the token map (e.g. $56\times56$ in stage 1 for a $224^2$ image)
- $C$: channels (token width)
- $4hwC^2$: q, k, v and output projections: the same for both
- $M$: window side; each query attends to $M^2=49$ keys instead of $hw$

The paper writes these as FLOPs; they are multiply-adds, as everywhere in this module. Global attention is quadratic in $hw$; window attention is linear, because each of the $hw/M^2$ windows costs $(M^2)^2C$ for the products. In stage 1 of Swin-T ($56\times56$ tokens, $C=96$) that is 2.00G multiply-adds per block for global attention versus 0.145G for windows, a 13.8× saving. The saving shrinks as the map shrinks: 2.7× in stage 2, 1.18× in stage 3, and none in stage 4, where the $7\times7$ map is a single window.

$$
\mathrm{Attention}(Q,K,V)=\mathrm{SoftMax}\!\Big(\frac{QK^\top}{\sqrt d}+B+\mathcal M\Big)V,\qquad B_{ij}=\hat B\big[\Delta y_{ij}+M-1,\;\Delta x_{ij}+M-1\big]
$$

- $Q,K,V\in\mathbb R^{M^2\times d}$: queries, keys and values of the tokens of one window, for one head
- $\hat B\in\mathbb R^{(2M-1)\times(2M-1)}$: learned **relative position bias** table per head ($13\times13=169$ entries for $M=7$), indexed by the offset $(\Delta y,\Delta x)\in[-6,6]^2$ between query and key
- $\mathcal M$: the shift mask: $0$ for allowed pairs, a large negative number ($-100$ in the official code) for pairs from different regions

Because the bias depends only on the offset, the same table is reused in every window, which gives the attention a translation-invariant prior, much like a convolution's weight sharing. In the gradient, a masked entry has softmax weight $\approx e^{-100}$, so $\partial L/\partial(QK^\top)_{ij}=a_{ij}\big(g_{ij}-\sum_k a_{ik}g_{ik}\big)\approx0$ there: masked pairs neither contribute forward nor receive gradient.

## Shifted windows: cyclic shift plus masking

With fixed windows, a token never sees anything outside its window, however deep the network. Swin alternates two layouts. Block $\ell$ uses regular windows (**W-MSA**). Block $\ell+1$ shifts the window grid by $\lfloor M/2\rfloor$ tokens in both directions (**SW-MSA**), so each new window straddles four old ones. Two blocks together connect neighbouring windows, and the receptive field keeps growing with depth, like stacked convolutions.

Shifting creates more windows, some smaller than $M\times M$: an $8\times8$ map with $M=4$ goes from $2\times2$ to $3\times3$ windows. Padding them to full size would cost $(\lceil h/M\rceil+1)^2$ windows. Swin instead **cyclically rolls** the map by $(-s,-s)$, so the partial windows at the far edges wrap around and pack into full $M\times M$ windows; the number of windows stays $(h/M)^2$. Tokens that were not adjacent before the roll now share a window, so a mask blocks attention between tokens from different regions. After attention the map is rolled back by $(+s,+s)$.

> Figure: Click a token to make it the query; the right matrix is its window's attention mask. On the default 8×8 map with M = 4, the shifted layout uses 4 rolled windows instead of the 9 a padded layout would need; the corner query (0,0) may attend to only 4 of the 16 tokens in its window (its own 2×2 region), and the mask blocks 448 of the 1,024 query-key pairs (43.8%). Toggle 'show after cyclic roll' to see the wrapped regions packed into one window. On the 14×14, M = 7 preset the share is 43.0%; for Swin-T's 56×56 stage-1 map only 18,240 of 153,664 pairs (11.9%) are masked, because just the last row and column of windows are split. The right panel counts Swin-T from its configuration: 28,288,354 parameters and 4.49 GMACs, matching torchvision's swin_t.

**Worked example: the shifted layout of an 8×8 map, M = 4, s = 2**

1. Roll by $(-2,-2)$. In rolled coordinates, label each row band $[0,4)$, $[4,6)$, $[6,8)$ and each column band the same way: $3\times3=9$ regions. These are exactly the slices the official code uses: $[0,h-M)$, $[h-M,h-s)$, $[h-s,h)$.
2. Partition the rolled map into the usual $2\times2$ windows of $4\times4$. The top-left window holds region 0 only: 16 tokens, no masking.
3. The top-right and bottom-left windows each hold two regions of 8 tokens: $2\cdot8\cdot8=128$ blocked ordered pairs each.
4. The bottom-right window holds four $2\times2$ regions: $16^2-4\cdot4^2=192$ blocked pairs. Total $0+128+128+192=448$ of $4\cdot16^2=1024$ pairs, 43.75%, as the figure reports.
5. The masked pairs are wasted compute, but they are cheap and the windows stay uniform, which batches well on GPUs. On large maps the waste is small: 11.9% at $56\times56$, $M=7$.

## The hierarchy

A $4\times4$ patch embedding (a stride-4 convolution, 48 → $C$ channels) gives a $56\times56$ token map. Each later stage starts with **patch merging**: concatenate each $2\times2$ group of tokens ($4C$ channels), apply LayerNorm and a linear layer $4C\to2C$. Resolution halves and width doubles, as in a CNN stage. Swin-T uses $C=96$ and $(2,2,6,2)$ blocks, with 3, 6, 12, 24 heads of width 32.

Swin-T stage by stage at 224² (counted by the figure's code; the sum matches torchvision's 28,288,354 params and 4.49 GFLOPS)

| stage | tokens | C | blocks | params | MACs |
| --- | --- | --- | --- | --- | --- |
| patch embed 4×4 | 56×56 | 96 | – | 4,896 | 14.5M |
| 1 | 56×56 | 96 | 2 | 224,694 | 752.6M |
| 2 (merge) | 28×28 | 192 | 2 | 966,252 | 780.9M |
| 3 (merge) | 14×14 | 384 | 6 | 10,955,400 | 2,183.0M |
| 4 (merge) | 7×7 | 768 | 2 | 15,366,576 | 758.8M |
| LN + head | 1 | 768 | – | 770,536 | 0.8M |

Swin variants on ImageNet-1K at 224², trained on ImageNet-1K only (Liu et al. 2021, Table 1a); params/MACs counted from the configs

| model | C | depths | params | MACs | top-1 |
| --- | --- | --- | --- | --- | --- |
| Swin-T | 96 | 2-2-6-2 | 28.3M | 4.49G | 81.3% |
| Swin-S | 96 | 2-2-18-2 | 49.6M | 8.74G | 83.0% |
| Swin-B | 128 | 2-2-18-2 | 87.8M | 15.43G | 83.5% |

The paper's ablation (Table 4) isolates the shift: on Swin-T, shifted windows raise ImageNet top-1 from 80.2% to 81.3%, with larger gains on COCO and ADE20K, where long-range context matters more. The relative position bias also beats absolute embeddings there.

```python
import torch
from torchvision.models import swin_t, Swin_T_Weights

def window_partition(x, M):                     # x: (B, H, W, C) -> (B*nW, M*M, C)
    B, H, W, C = x.shape
    x = x.view(B, H // M, M, W // M, M, C).permute(0, 1, 3, 2, 4, 5)
    return x.reshape(-1, M * M, C)

def shift_mask(H, W, M, s):                     # (nW, M*M, M*M) additive mask
    img = torch.zeros(1, H, W, 1)
    cnt = 0
    for hs in (slice(0, -M), slice(-M, -s), slice(-s, None)):
        for ws in (slice(0, -M), slice(-M, -s), slice(-s, None)):
            img[:, hs, ws, :] = cnt; cnt += 1
    lab = window_partition(img, M).squeeze(-1)  # region label of every token, per window
    diff = lab[:, None, :] - lab[:, :, None]
    return diff.masked_fill(diff != 0, -100.0).masked_fill(diff == 0, 0.0)

H = W = 8; M = 4; s = M // 2
mask = shift_mask(H, W, M, s)
print((mask != 0).sum().item(), mask.numel())  # 448 1024

x = torch.randn(1, H, W, 96)
x = torch.roll(x, shifts=(-s, -s), dims=(1, 2))   # cyclic shift before SW-MSA
win = window_partition(x, M)                       # (4, 16, 96): 4 windows, not 9
# ... attention(win) + mask per window ..., then roll back with shifts=(s, s)

model = swin_t(weights=Swin_T_Weights.IMAGENET1K_V1).eval()
print(sum(p.numel() for p in model.parameters()))  # 28288354
feats = model.features(torch.randn(1, 3, 224, 224))
print(feats.shape)                                 # torch.Size([1, 7, 7, 768]), channels-last
```

Window partition and the shift mask from scratch, then torchvision's swin_t

> **Resolution must fit the windows:** Every stage's map must be divisible into $M\times M$ windows, otherwise it must be padded (torchvision pads and then crops). With $M=7$ and a stride-32 last stage, inputs that are multiples of 224 are the clean case; Swin at 384 uses $M=12$. When a map is no larger than the window, there is nothing to shift: the official implementation sets the shift to 0 and the window to the map size, as in Swin-T's stage 4.

> **Back towards convolution:** Window attention is local, shares its relative-position bias across windows, and is stacked in a downsampling pyramid: three properties of a CNN. The difference is that the mixing weights are computed from the content of the window rather than fixed. **ConvNeXt** took the reverse path, modernizing a ResNet toward Swin's design, and matched Swin with pure convolutions, which suggested the hierarchy and training recipe matter as much as attention itself.

> **Swin V2:** Liu et al. (CVPR 2022) scaled Swin to 3 billion parameters and $1536^2$ inputs. The changes were residual post-normalization, scaled cosine attention instead of the dot product, and a log-spaced continuous position bias produced by a small MLP, so that window sizes can change between pre-training and fine-tuning. torchvision ships `swin_v2_t/s/b`.

**Strengths**

- Attention cost linear in image area: practical for detection and segmentation at high resolution
- Feature pyramid at strides 4–32: drop-in replacement for a ResNet backbone in FPN, Mask R-CNN, UperNet
- Shifted windows connect neighbours without overlapping windows or extra windows
- Better accuracy per FLOP than ViT/DeiT on ImageNet, much better on dense tasks

**Limitations**

- More complex than ViT: rolls, masks, padding, relative position index tables
- Window partition, roll and mask cost memory traffic; real speed lags the FLOP count, especially on non-GPU hardware
- Receptive field grows only gradually; global context needs depth
- Matched by a pure ConvNet (ConvNeXt) with the same budget and recipe

**Question:** Stage 1 of Swin-T has a 56×56 token map with M = 7. How many windows does a shifted-window block use with the cyclic-shift trick, and how many would naive padding need?

- A. 64 and 64
- B. 64 and 81
- C. 49 and 64
- D. 81 and 81

**Answer:** B. 64 and 81 — Regular partition: $(56/7)^2=64$ windows. Shifting by 3 creates partial windows on every border, $9\times9=81$ if each is padded to $7\times7$. Rolling packs the partial windows back into 64 full ones, and the mask keeps the wrapped regions from attending to each other.

**Question:** Why does Swin's attention cost grow linearly, not quadratically, with the number of tokens?

- A. It uses linear attention without a softmax
- B. Each query attends only to the M² tokens of its own window, so the products cost 2M²·hw·C
- C. Patch merging removes tokens in every block
- D. The relative position bias replaces the QKᵀ product

**Answer:** B. Each query attends only to the M² tokens of its own window, so the products cost 2M²·hw·C — The softmax attention is standard, just restricted to windows. With a fixed window size $M$, each of the $hw$ queries costs $O(M^2C)$, so the total is $O(M^2hwC)$, linear in $hw$. Global attention costs $O((hw)^2C)$.

**Key points**

- Self-attention within $M\times M$ windows ($M=7$): cost $4hwC^2+2M^2hwC$, linear in image area.
- Alternate regular and shifted ($\lfloor M/2\rfloor$) windows so information crosses window borders.
- Cyclic shift + mask keeps the number of windows at $(h/M)^2$ instead of $(h/M+1)^2$; masked pairs get $-100$ before the softmax.
- Relative position bias: a learned $(2M-1)^2$ table per head, shared by all windows.
- Patch merging (2×2 concat, LN, linear 4C→2C) builds a ResNet-like pyramid; Swin-T is 28.3M params, 4.49 GMACs, 81.3% top-1.

---

From MLFoundry (https://www.mlfoundry.in). The web page adds interactive figures and diagrams.
