Definition
Swin Transformer (Shifted window Transformer) is a vision Transformer that computes self-attention only within non-overlapping windows of tokens (), alternates between a regular and a shifted window layout in consecutive blocks so that neighbouring windows exchange information, and builds a four-stage hierarchy by merging neighbouring tokens between stages. Its feature maps have strides 4, 8, 16, 32, exactly like a ResNet's.
Liu et al. introduced Swin in "Swin Transformer: Hierarchical Vision Transformer using Shifted Windows" (ICCV 2021, Marr Prize). They targeted two problems of Vision Transformers (ViT) as general backbones. First, global attention costs in the number of tokens, which is prohibitive for detection and segmentation at high resolution. Second, ViT keeps one coarse scale ( downsampled) throughout, while dense prediction wants a feature pyramid. Swin reintroduces two convolutional priors, locality and hierarchy, while keeping attention as the mixing operator.
It became the default Transformer backbone for dense prediction: Swin-L reached 58.7 box AP and 51.1 mask AP on COCO test-dev and 53.5 mIoU on ADE20K, state of the art at publication. On ImageNet-1K classification, Swin-T reached 81.3% top-1 at 4.5 GFLOPs, against 79.8% for DeiT-S at a similar cost (paper Table 1).
Window attention and its cost
- the token map (e.g. in stage 1 for a image)
- channels (token width)
- q, k, v and output projections: the same for both
- window side; each query attends to keys instead of
The paper writes these as FLOPs; they are multiply-adds, as everywhere in this module. Global attention is quadratic in ; window attention is linear, because each of the windows costs for the products. In stage 1 of Swin-T ( tokens, ) that is 2.00G multiply-adds per block for global attention versus 0.145G for windows, a 13.8× saving. The saving shrinks as the map shrinks: 2.7× in stage 2, 1.18× in stage 3, and none in stage 4, where the map is a single window.
- queries, keys and values of the tokens of one window, for one head
- learned relative position bias table per head ( entries for ), indexed by the offset between query and key
- the shift mask: for allowed pairs, a large negative number ( in the official code) for pairs from different regions
Because the bias depends only on the offset, the same table is reused in every window, which gives the attention a translation-invariant prior, much like a convolution's weight sharing. In the gradient, a masked entry has softmax weight , so there: masked pairs neither contribute forward nor receive gradient.
Shifted windows: cyclic shift plus masking
With fixed windows, a token never sees anything outside its window, however deep the network. Swin alternates two layouts. Block uses regular windows (W-MSA). Block shifts the window grid by tokens in both directions (SW-MSA), so each new window straddles four old ones. Two blocks together connect neighbouring windows, and the receptive field keeps growing with depth, like stacked convolutions.
Shifting creates more windows, some smaller than : an map with goes from to windows. Padding them to full size would cost windows. Swin instead cyclically rolls the map by , so the partial windows at the far edges wrap around and pack into full windows; the number of windows stays . Tokens that were not adjacent before the roll now share a window, so a mask blocks attention between tokens from different regions. After attention the map is rolled back by .
Worked example: the shifted layout of an 8×8 map, M = 4, s = 2
- 1Roll by . In rolled coordinates, label each row band , , and each column band the same way: regions. These are exactly the slices the official code uses: , , .
- 2Partition the rolled map into the usual windows of . The top-left window holds region 0 only: 16 tokens, no masking.
- 3The top-right and bottom-left windows each hold two regions of 8 tokens: blocked ordered pairs each.
- 4The bottom-right window holds four regions: blocked pairs. Total of pairs, 43.75%, as the figure reports.
- 5The masked pairs are wasted compute, but they are cheap and the windows stay uniform, which batches well on GPUs. On large maps the waste is small: 11.9% at , .
The hierarchy
A patch embedding (a stride-4 convolution, 48 → channels) gives a token map. Each later stage starts with patch merging: concatenate each group of tokens ( channels), apply LayerNorm and a linear layer . Resolution halves and width doubles, as in a CNN stage. Swin-T uses and blocks, with 3, 6, 12, 24 heads of width 32.
| stage | tokens | C | blocks | params | MACs |
|---|---|---|---|---|---|
| patch embed 4×4 | 56×56 | 96 | – | 4,896 | 14.5M |
| 1 | 56×56 | 96 | 2 | 224,694 | 752.6M |
| 2 (merge) | 28×28 | 192 | 2 | 966,252 | 780.9M |
| 3 (merge) | 14×14 | 384 | 6 | 10,955,400 | 2,183.0M |
| 4 (merge) | 7×7 | 768 | 2 | 15,366,576 | 758.8M |
| LN + head | 1 | 768 | – | 770,536 | 0.8M |
| model | C | depths | params | MACs | top-1 |
|---|---|---|---|---|---|
| Swin-T | 96 | 2-2-6-2 | 28.3M | 4.49G | 81.3% |
| Swin-S | 96 | 2-2-18-2 | 49.6M | 8.74G | 83.0% |
| Swin-B | 128 | 2-2-18-2 | 87.8M | 15.43G | 83.5% |
The paper's ablation (Table 4) isolates the shift: on Swin-T, shifted windows raise ImageNet top-1 from 80.2% to 81.3%, with larger gains on COCO and ADE20K, where long-range context matters more. The relative position bias also beats absolute embeddings there.
1import torch2from torchvision.models import swin_t, Swin_T_Weights34def window_partition(x, M): # x: (B, H, W, C) -> (B*nW, M*M, C)5 B, H, W, C = x.shape6 x = x.view(B, H // M, M, W // M, M, C).permute(0, 1, 3, 2, 4, 5)7 return x.reshape(-1, M * M, C)89def shift_mask(H, W, M, s): # (nW, M*M, M*M) additive mask10 img = torch.zeros(1, H, W, 1)11 cnt = 012 for hs in (slice(0, -M), slice(-M, -s), slice(-s, None)):13 for ws in (slice(0, -M), slice(-M, -s), slice(-s, None)):14 img[:, hs, ws, :] = cnt; cnt += 115 lab = window_partition(img, M).squeeze(-1) # region label of every token, per window16 diff = lab[:, None, :] - lab[:, :, None]17 return diff.masked_fill(diff != 0, -100.0).masked_fill(diff == 0, 0.0)1819H = W = 8; M = 4; s = M // 220mask = shift_mask(H, W, M, s)21print((mask != 0).sum().item(), mask.numel()) # 448 10242223x = torch.randn(1, H, W, 96)24x = torch.roll(x, shifts=(-s, -s), dims=(1, 2)) # cyclic shift before SW-MSA25win = window_partition(x, M) # (4, 16, 96): 4 windows, not 926# ... attention(win) + mask per window ..., then roll back with shifts=(s, s)2728model = swin_t(weights=Swin_T_Weights.IMAGENET1K_V1).eval()29print(sum(p.numel() for p in model.parameters())) # 2828835430feats = model.features(torch.randn(1, 3, 224, 224))31print(feats.shape) # torch.Size([1, 7, 7, 768]), channels-lastUse when
- +Attention cost linear in image area: practical for detection and segmentation at high resolution
- +Feature pyramid at strides 4–32: drop-in replacement for a ResNet backbone in FPN, Mask R-CNN, UperNet
- +Shifted windows connect neighbours without overlapping windows or extra windows
- +Better accuracy per FLOP than ViT/DeiT on ImageNet, much better on dense tasks
Avoid when
- −More complex than ViT: rolls, masks, padding, relative position index tables
- −Window partition, roll and mask cost memory traffic; real speed lags the FLOP count, especially on non-GPU hardware
- −Receptive field grows only gradually; global context needs depth
- −Matched by a pure ConvNet (ConvNeXt) with the same budget and recipe
Check yourself
Stage 1 of Swin-T has a 56×56 token map with M = 7. How many windows does a shifted-window block use with the cyclic-shift trick, and how many would naive padding need?
Check yourself
Why does Swin's attention cost grow linearly, not quadratically, with the number of tokens?
Remember
- ✓Self-attention within windows (): cost , linear in image area.
- ✓Alternate regular and shifted () windows so information crosses window borders.
- ✓Cyclic shift + mask keeps the number of windows at instead of ; masked pairs get before the softmax.
- ✓Relative position bias: a learned table per head, shared by all windows.
- ✓Patch merging (2×2 concat, LN, linear 4C→2C) builds a ResNet-like pyramid; Swin-T is 28.3M params, 4.49 GMACs, 81.3% top-1.