Swin Transformer

A hierarchical vision Transformer: self-attention inside 7×7 windows (cost linear in image size), windows shifted every other block so information crosses their borders, and patch merging that halves resolution like a CNN stage. Swin-T: 28M params, 4.5 GFLOPs, 81.3%.

14 min read992 wordsadvanced0 viewsupdated 2026-10-11

Definition

Swin Transformer (Shifted window Transformer) is a vision Transformer that computes self-attention only within non-overlapping M×MM\times M windows of tokens (M=7M=7), alternates between a regular and a shifted window layout in consecutive blocks so that neighbouring windows exchange information, and builds a four-stage hierarchy by merging 2×22\times2 neighbouring tokens between stages. Its feature maps have strides 4, 8, 16, 32, exactly like a ResNet's.

Liu et al. introduced Swin in "Swin Transformer: Hierarchical Vision Transformer using Shifted Windows" (ICCV 2021, Marr Prize). They targeted two problems of Vision Transformers (ViT) as general backbones. First, global attention costs O(N2)O(N^2) in the number of tokens, which is prohibitive for detection and segmentation at high resolution. Second, ViT keeps one coarse scale (16×16\times downsampled) throughout, while dense prediction wants a feature pyramid. Swin reintroduces two convolutional priors, locality and hierarchy, while keeping attention as the mixing operator.

It became the default Transformer backbone for dense prediction: Swin-L reached 58.7 box AP and 51.1 mask AP on COCO test-dev and 53.5 mIoU on ADE20K, state of the art at publication. On ImageNet-1K classification, Swin-T reached 81.3% top-1 at 4.5 GFLOPs, against 79.8% for DeiT-S at a similar cost (paper Table 1).

Window attention and its cost

Ω(MSA)=4hwC2+2(hw)2C,Ω(W-MSA)=4hwC2+2M2 hw C\Omega(\mathrm{MSA})=4hwC^2+2(hw)^2C,\qquad \Omega(\mathrm{W\text{-}MSA})=4hwC^2+2M^2\,hw\,C
h×wh\times w
the token map (e.g. 56×5656\times56 in stage 1 for a 2242224^2 image)
CC
channels (token width)
4hwC24hwC^2
q, k, v and output projections: the same for both
MM
window side; each query attends to M2=49M^2=49 keys instead of hwhw

The paper writes these as FLOPs; they are multiply-adds, as everywhere in this module. Global attention is quadratic in hwhw; window attention is linear, because each of the hw/M2hw/M^2 windows costs (M2)2C(M^2)^2C for the products. In stage 1 of Swin-T (56×5656\times56 tokens, C=96C=96) that is 2.00G multiply-adds per block for global attention versus 0.145G for windows, a 13.8× saving. The saving shrinks as the map shrinks: 2.7× in stage 2, 1.18× in stage 3, and none in stage 4, where the 7×77\times7 map is a single window.

Attention(Q,K,V)=SoftMax ⁣(QK⊤d+B+M)V,Bij=B^[Δyij+M−1,  Δxij+M−1]\mathrm{Attention}(Q,K,V)=\mathrm{SoftMax}\!\Big(\frac{QK^\top}{\sqrt d}+B+\mathcal M\Big)V,\qquad B_{ij}=\hat B\big[\Delta y_{ij}+M-1,\;\Delta x_{ij}+M-1\big]
Q,K,V∈RM2×dQ,K,V\in\mathbb R^{M^2\times d}
queries, keys and values of the tokens of one window, for one head
B^∈R(2M−1)×(2M−1)\hat B\in\mathbb R^{(2M-1)\times(2M-1)}
learned relative position bias table per head (13×13=16913\times13=169 entries for M=7M=7), indexed by the offset (Δy,Δx)∈[−6,6]2(\Delta y,\Delta x)\in[-6,6]^2 between query and key
M\mathcal M
the shift mask: 00 for allowed pairs, a large negative number (−100-100 in the official code) for pairs from different regions

Because the bias depends only on the offset, the same table is reused in every window, which gives the attention a translation-invariant prior, much like a convolution's weight sharing. In the gradient, a masked entry has softmax weight ≈e−100\approx e^{-100}, so ∂L/∂(QK⊤)ij=aij(gij−∑kaikgik)≈0\partial L/\partial(QK^\top)_{ij}=a_{ij}\big(g_{ij}-\sum_k a_{ik}g_{ik}\big)\approx0 there: masked pairs neither contribute forward nor receive gradient.

Shifted windows: cyclic shift plus masking

With fixed windows, a token never sees anything outside its window, however deep the network. Swin alternates two layouts. Block ℓ\ell uses regular windows (W-MSA). Block ℓ+1\ell+1 shifts the window grid by ⌊M/2⌋\lfloor M/2\rfloor tokens in both directions (SW-MSA), so each new window straddles four old ones. Two blocks together connect neighbouring windows, and the receptive field keeps growing with depth, like stacked convolutions.

Shifting creates more windows, some smaller than M×MM\times M: an 8×88\times8 map with M=4M=4 goes from 2×22\times2 to 3×33\times3 windows. Padding them to full size would cost (⌈h/M⌉+1)2(\lceil h/M\rceil+1)^2 windows. Swin instead cyclically rolls the map by (−s,−s)(-s,-s), so the partial windows at the far edges wrap around and pack into full M×MM\times M windows; the number of windows stays (h/M)2(h/M)^2. Tokens that were not adjacent before the roll now share a window, so a mask blocks attention between tokens from different regions. After attention the map is rolled back by (+s,+s)(+s,+s).

8×8 tokens, M=4: SW-MSA, windows shifted by 2━ window edge ┄ mask edge ✕ masked keyquery (0,0) attends to 4 of 16 tokens in its windowwindows: 4 (rolled) vs 9 if paddedmasked pairs 448 / 1024 (43.8%)mask of this window (16×16)■ 0 added ■ −100 addedbright row = the querySwin-T, 224²C=96, depths 2-2-6-228,288,354 params4.49 GMACspaper: 28M, 4.5GIN-1K top-1 81.3%one block's attention MACsst map global window1 56²×96 2.00G 145M2 28²×192 352M 130M3 14²×384 145M 123M4 7²×768 119M 119MSwin-T MACs per stage (map halves, C doubles)embed 4×414Mstage 1753Mstage 2781Mstage 32.18Gstage 4759MLN + head1M
layout
map
modelclick a token to make it the query
fig.Click a token to make it the query; the right matrix is its window's attention mask. On the default 8×8 map with M = 4, the shifted layout uses 4 rolled windows instead of the 9 a padded layout would need; the corner query (0,0) may attend to only 4 of the 16 tokens in its window (its own 2×2 region), and the mask blocks 448 of the 1,024 query-key pairs (43.8%). Toggle 'show after cyclic roll' to see the wrapped regions packed into one window. On the 14×14, M = 7 preset the share is 43.0%; for Swin-T's 56×56 stage-1 map only 18,240 of 153,664 pairs (11.9%) are masked, because just the last row and column of windows are split. The right panel counts Swin-T from its configuration: 28,288,354 parameters and 4.49 GMACs, matching torchvision's swin_t.

Worked example: the shifted layout of an 8×8 map, M = 4, s = 2

  1. 1Roll by (−2,−2)(-2,-2). In rolled coordinates, label each row band [0,4)[0,4), [4,6)[4,6), [6,8)[6,8) and each column band the same way: 3×3=93\times3=9 regions. These are exactly the slices the official code uses: [0,h−M)[0,h-M), [h−M,h−s)[h-M,h-s), [h−s,h)[h-s,h).
  2. 2Partition the rolled map into the usual 2×22\times2 windows of 4×44\times4. The top-left window holds region 0 only: 16 tokens, no masking.
  3. 3The top-right and bottom-left windows each hold two regions of 8 tokens: 2⋅8⋅8=1282\cdot8\cdot8=128 blocked ordered pairs each.
  4. 4The bottom-right window holds four 2×22\times2 regions: 162−4⋅42=19216^2-4\cdot4^2=192 blocked pairs. Total 0+128+128+192=4480+128+128+192=448 of 4⋅162=10244\cdot16^2=1024 pairs, 43.75%, as the figure reports.
  5. 5The masked pairs are wasted compute, but they are cheap and the windows stay uniform, which batches well on GPUs. On large maps the waste is small: 11.9% at 56×5656\times56, M=7M=7.

The hierarchy

A 4×44\times4 patch embedding (a stride-4 convolution, 48 → CC channels) gives a 56×5656\times56 token map. Each later stage starts with patch merging: concatenate each 2×22\times2 group of tokens (4C4C channels), apply LayerNorm and a linear layer 4C→2C4C\to2C. Resolution halves and width doubles, as in a CNN stage. Swin-T uses C=96C=96 and (2,2,6,2)(2,2,6,2) blocks, with 3, 6, 12, 24 heads of width 32.

stagetokensCblocksparamsMACs
patch embed 4×456×5696–4,89614.5M
156×56962224,694752.6M
2 (merge)28×281922966,252780.9M
3 (merge)14×14384610,955,4002,183.0M
4 (merge)7×7768215,366,576758.8M
LN + head1768–770,5360.8M
Swin-T stage by stage at 224² (counted by the figure's code; the sum matches torchvision's 28,288,354 params and 4.49 GFLOPS)
modelCdepthsparamsMACstop-1
Swin-T962-2-6-228.3M4.49G81.3%
Swin-S962-2-18-249.6M8.74G83.0%
Swin-B1282-2-18-287.8M15.43G83.5%
Swin variants on ImageNet-1K at 224², trained on ImageNet-1K only (Liu et al. 2021, Table 1a); params/MACs counted from the configs

The paper's ablation (Table 4) isolates the shift: on Swin-T, shifted windows raise ImageNet top-1 from 80.2% to 81.3%, with larger gains on COCO and ADE20K, where long-range context matters more. The relative position bias also beats absolute embeddings there.

python
1import torch
2from torchvision.models import swin_t, Swin_T_Weights
3
4def window_partition(x, M): # x: (B, H, W, C) -> (B*nW, M*M, C)
5 B, H, W, C = x.shape
6 x = x.view(B, H // M, M, W // M, M, C).permute(0, 1, 3, 2, 4, 5)
7 return x.reshape(-1, M * M, C)
8
9def shift_mask(H, W, M, s): # (nW, M*M, M*M) additive mask
10 img = torch.zeros(1, H, W, 1)
11 cnt = 0
12 for hs in (slice(0, -M), slice(-M, -s), slice(-s, None)):
13 for ws in (slice(0, -M), slice(-M, -s), slice(-s, None)):
14 img[:, hs, ws, :] = cnt; cnt += 1
15 lab = window_partition(img, M).squeeze(-1) # region label of every token, per window
16 diff = lab[:, None, :] - lab[:, :, None]
17 return diff.masked_fill(diff != 0, -100.0).masked_fill(diff == 0, 0.0)
18
19H = W = 8; M = 4; s = M // 2
20mask = shift_mask(H, W, M, s)
21print((mask != 0).sum().item(), mask.numel()) # 448 1024
22
23x = torch.randn(1, H, W, 96)
24x = torch.roll(x, shifts=(-s, -s), dims=(1, 2)) # cyclic shift before SW-MSA
25win = window_partition(x, M) # (4, 16, 96): 4 windows, not 9
26# ... attention(win) + mask per window ..., then roll back with shifts=(s, s)
27
28model = swin_t(weights=Swin_T_Weights.IMAGENET1K_V1).eval()
29print(sum(p.numel() for p in model.parameters())) # 28288354
30feats = model.features(torch.randn(1, 3, 224, 224))
31print(feats.shape) # torch.Size([1, 7, 7, 768]), channels-last
Window partition and the shift mask from scratch, then torchvision's swin_t

Use when

  • +Attention cost linear in image area: practical for detection and segmentation at high resolution
  • +Feature pyramid at strides 4–32: drop-in replacement for a ResNet backbone in FPN, Mask R-CNN, UperNet
  • +Shifted windows connect neighbours without overlapping windows or extra windows
  • +Better accuracy per FLOP than ViT/DeiT on ImageNet, much better on dense tasks

Avoid when

  • −More complex than ViT: rolls, masks, padding, relative position index tables
  • −Window partition, roll and mask cost memory traffic; real speed lags the FLOP count, especially on non-GPU hardware
  • −Receptive field grows only gradually; global context needs depth
  • −Matched by a pure ConvNet (ConvNeXt) with the same budget and recipe

Check yourself

Stage 1 of Swin-T has a 56×56 token map with M = 7. How many windows does a shifted-window block use with the cyclic-shift trick, and how many would naive padding need?

Check yourself

Why does Swin's attention cost grow linearly, not quadratically, with the number of tokens?

Remember

  • ✓Self-attention within M×MM\times M windows (M=7M=7): cost 4hwC2+2M2hwC4hwC^2+2M^2hwC, linear in image area.
  • ✓Alternate regular and shifted (⌊M/2⌋\lfloor M/2\rfloor) windows so information crosses window borders.
  • ✓Cyclic shift + mask keeps the number of windows at (h/M)2(h/M)^2 instead of (h/M+1)2(h/M+1)^2; masked pairs get −100-100 before the softmax.
  • ✓Relative position bias: a learned (2M−1)2(2M-1)^2 table per head, shared by all windows.
  • ✓Patch merging (2×2 concat, LN, linear 4C→2C) builds a ResNet-like pyramid; Swin-T is 28.3M params, 4.49 GMACs, 81.3% top-1.
← back to Deep Learning roadmap