Definition
Swish is the activation , the input multiplied by a sigmoid of itself. is a fixed constant or a learned parameter. With it is called SiLU (sigmoid-weighted linear unit), .
Swish is self-gated. An LSTM or GLU gate multiplies a signal by a sigmoid of a separate projection. Swish uses the same scalar for both, so it needs no extra parameters and drops in wherever a ReLU was.
It is an unusual case of an activation found by search. Ramachandran, Zoph & Le (2017, "Searching for Activation Functions") used reinforcement learning to search over compositions of simple unary and binary functions, and was the best they found. Their abstract reports that replacing ReLU with Swish raised ImageNet top-1 accuracy by 0.9% for Mobile NASNet-A and 0.6% for Inception-ResNet-v2.
The function was already known. Hendrycks & Gimpel (2016) named it SiLU as a cheaper relative of GELU, and Elfwing, Uchibe & Doya (2017) used it under the same name for reinforcement learning. PyTorch calls it nn.SiLU.
Formula and derivative
- logistic sigmoid,
- gate sharpness; is SiLU, and may be trained per layer or per channel
- the gate value: what the slope would be if the gate were frozen
- change of the gate with x; negative for x < 0, which creates the dip
- itself: the second form reuses the forward output, which is how fused kernels compute the backward pass
At the slope is for every . As for GELU, , so .
One knob from linear to ReLU
Substitute : . Every member of the family is the SiLU curve shrunk by along both axes. Two limits follow:
- for every input: a linear function, no nonlinearity at all
- the gate becomes the step : ReLU
The shrinking also controls the dip. SiLU's minimum is at , the root of , with value . So bottoms out at with depth . That depth is also its largest distance from ReLU anywhere on the real line. The slope ranges from (at ) to (at ), whatever is.
At a glance (β = 1, SiLU)
- Range
- ; in general
- Monotonic
- No. Decreasing on
- Smooth
- Yes, infinitely differentiable (for finite ); slope in
- Zero-centred
- No. For the output has mean and std
- Saturation
- Below only: and as ; unbounded above
- Cost
- One exp and one division per element; hard-swish replaces them with a clamp
Hard-swish
On mobile hardware the exponential is slow and awkward to quantize. MobileNetV3 (Howard et al., 2019) replaced the sigmoid with a piecewise-linear ramp:
- a "hard sigmoid": 0 below , 1 above , linear in between
It is exactly 0 for and exactly for . Its largest deviation from SiLU is , reached at , and its minimum is at , deeper than SiLU's. It is piecewise quadratic, so its slope is continuous but has corners.
Worked example
| x | σ(x) | SiLU(x) | SiLU′(x) | h-swish(x) |
|---|---|---|---|---|
| −3 | 0.0474 | −0.1423 | −0.0881 | 0.0000 |
| −2 | 0.1192 | −0.2384 | −0.0908 | −0.3333 |
| −1.2785 | 0.2178 | −0.2785 | 0.0000 | −0.3668 |
| −1 | 0.2689 | −0.2689 | 0.0723 | −0.3333 |
| −0.5 | 0.3775 | −0.1888 | 0.2600 | −0.2083 |
| 0 | 0.5000 | 0.0000 | 0.5000 | 0.0000 |
| 1 | 0.7311 | 0.7311 | 0.9277 | 0.6667 |
| 2 | 0.8808 | 1.7616 | 1.0908 | 1.6667 |
| 3 | 0.9526 | 2.8577 | 1.0881 | 3.0000 |
Check one row by hand. At : , so , and . The same input through ReLU gives output 0 and slope 0.
Effect on training, and where it is used
Like GELU, Swish passes a small gradient for negative inputs (no dead units), is smooth, and has a ReLU-like slope near 1 for large positive inputs, so deep stacks behave much like ReLU stacks. For unit-variance inputs against ReLU's , so a plain stack with He initialization starts with slightly shrinking activations. Normalization layers make this irrelevant in most modern architectures.
Where it's used
EfficientNet
Tan & Le (2019) use Swish () throughout the EfficientNet family.
MobileNetV3
Hard-swish mainly in the second half of the network, where the paper found it pays off, with ReLU in most early layers.
LLMs via SwiGLU
LLaMA and PaLM use a feed-forward block (SwiGLU, Shazeer 2020). There SiLU gates a separate linear projection instead of itself.
YOLO detectors
Later YOLOv5 releases and several successors use nn.SiLU as their default activation.
1import torch, torch.nn as nn, torch.nn.functional as F23x = torch.tensor([-2., -1., 0., 1., 2.], requires_grad=True)4print(nn.SiLU()(x)) # [-0.2384, -0.2689, 0.0000, 0.7311, 1.7616]5print(F.hardswish(x)) # [-0.3333, -0.3333, 0.0000, 0.6667, 1.6667]67class Swish(nn.Module): # from scratch, β learned (one per layer)8 def __init__(self, beta=1.0):9 super().__init__()10 self.beta = nn.Parameter(torch.tensor(beta))11 def forward(self, x):12 return x * torch.sigmoid(self.beta * x)1314Swish()(x).sum().backward()15print(x.grad) # [-0.0908, 0.0723, 0.5000, 0.9277, 1.0908]Use when
- +Drop-in ReLU replacement with no extra parameters (at fixed β).
- +Smooth, non-zero gradient for negative inputs.
- +β interpolates between linear and ReLU, and the hard-swish form is cheap enough for mobile.
- +SiLU is the gate inside SwiGLU, the standard LLM feed-forward block.
Avoid when
- −An exp and a division per element: slower than ReLU when not fused.
- −Not monotonic and not zero-centred.
- −Gains over ReLU are modest (fractions of a percent in the original paper) and do not appear in every setting.
Check yourself
You double β. What happens to the location and depth of Swish's negative minimum?
Check yourself
What is the slope of x·σ(βx) at x = 0?
Remember
- ✓; with it is SiLU (Hendrycks & Gimpel 2016; Elfwing et al. 2017), and Ramachandran et al. found it by automated search (2017).
- ✓, for every .
- ✓ gives , gives ReLU; the minimum at scales as .
- ✓Hard-swish (MobileNetV3) is within of SiLU.
- ✓Used in EfficientNet, MobileNetV3 (hard-swish) and LLaMA/PaLM through SwiGLU.