Swish

x·σ(βx): the input gates itself through a sigmoid. One knob β slides it from the line x/2 to ReLU; at β = 1 it is the SiLU used in EfficientNet and LLaMA.

8 min read570 wordscore0 viewsupdated 2026-10-11

Definition

Swish is the activation swishβ(x)=x σ(βx)\mathrm{swish}_\beta(x)=x\,\sigma(\beta x), the input multiplied by a sigmoid of itself. β\beta is a fixed constant or a learned parameter. With β=1\beta=1 it is called SiLU (sigmoid-weighted linear unit), x σ(x)x\,\sigma(x).

Swish is self-gated. An LSTM or GLU gate multiplies a signal by a sigmoid of a separate projection. Swish uses the same scalar for both, so it needs no extra parameters and drops in wherever a ReLU was.

It is an unusual case of an activation found by search. Ramachandran, Zoph & Le (2017, "Searching for Activation Functions") used reinforcement learning to search over compositions of simple unary and binary functions, and x σ(βx)x\,\sigma(\beta x) was the best they found. Their abstract reports that replacing ReLU with Swish raised ImageNet top-1 accuracy by 0.9% for Mobile NASNet-A and 0.6% for Inception-ResNet-v2.

The β=1\beta=1 function was already known. Hendrycks & Gimpel (2016) named it SiLU as a cheaper relative of GELU, and Elfwing, Uchibe & Doya (2017) used it under the same name for reinforcement learning. PyTorch calls it nn.SiLU.

Formula and derivative

swishβ(x)=x σ(βx)=x1+e−βx\mathrm{swish}_\beta(x)=x\,\sigma(\beta x)=\frac{x}{1+e^{-\beta x}}
σ\sigma
logistic sigmoid, σ(t)=1/(1+e−t)\sigma(t)=1/(1+e^{-t})
β\beta
gate sharpness; β=1\beta=1 is SiLU, and β\beta may be trained per layer or per channel
swishβ′(x)=σ(βx)+βx σ(βx)(1−σ(βx))=βf(x)+σ(βx)(1−βf(x))\mathrm{swish}_\beta'(x)=\sigma(\beta x)+\beta x\,\sigma(\beta x)\big(1-\sigma(\beta x)\big)=\beta f(x)+\sigma(\beta x)\big(1-\beta f(x)\big)
σ(βx)\sigma(\beta x)
the gate value: what the slope would be if the gate were frozen
βx σ(1−σ)\beta x\,\sigma(1-\sigma)
change of the gate with x; negative for x < 0, which creates the dip
f(x)f(x)
swishβ(x)\mathrm{swish}_\beta(x) itself: the second form reuses the forward output, which is how fused kernels compute the backward pass

At x=0x=0 the slope is σ(0)=12\sigma(0)=\tfrac12 for every β\beta. As for GELU, f(x)−f(−x)=xf(x)-f(-x)=x, so f′(x)+f′(−x)=1f'(x)+f'(-x)=1.

One knob from linear to ReLU

Substitute u=βxu=\beta x: swishβ(x)=SiLU(βx)/β\mathrm{swish}_\beta(x)=\mathrm{SiLU}(\beta x)/\beta. Every member of the family is the SiLU curve shrunk by 1/β1/\beta along both axes. Two limits follow:

β→0:  swishβ(x)→x2,β→∞:  swishβ(x)→max⁡(0,x)\beta\to0:\ \ \mathrm{swish}_\beta(x)\to\frac{x}{2},\qquad \beta\to\infty:\ \ \mathrm{swish}_\beta(x)\to\max(0,x)
x/2x/2
σ(0)=12\sigma(0)=\tfrac12 for every input: a linear function, no nonlinearity at all
max⁡(0,x)\max(0,x)
the gate becomes the step 1[x>0]\mathbb{1}[x>0]: ReLU

The shrinking also controls the dip. SiLU's minimum is at u∗=−1.2785u^*=-1.2785, the root of 1+u(1−σ(u))=01+u(1-\sigma(u))=0, with value −0.2785-0.2785. So swishβ\mathrm{swish}_\beta bottoms out at x=−1.2785/βx=-1.2785/\beta with depth −0.2785/β-0.2785/\beta. That depth is also its largest distance from ReLU anywhere on the real line. The slope ranges from −0.0998-0.0998 (at x=−2.3994/βx=-2.3994/\beta) to 1.09981.0998 (at x=+2.3994/βx=+2.3994/\beta), whatever β\beta is.

swish_β(x) = x·σ(βx), β = 1.00-2-101234-4-2024β→0: x/2β→∞: ReLUmin -0.278 at x = -1.278max gap on [−4, 4] vs β(log–log)1e-31e-21e-1110β=0.1β=1β=10to ReLU = 0.278 (= 0.2785/β)to x/2 = 1.928f′(0) = 0.50 for every βpeak f′ 1.100 at x = 2.40— current β — β = 0.25, 0.5, 1, 2, 4, 10
presets
minimum -0.2785 at x = -1.2785both scale as 1/β
fig.Drag β. Faint curves are fixed members of the family. The dot tracks the negative minimum, which moves toward 0 and gets shallower as 1/β. On the right, the distance to ReLU (rose) falls as 0.2785/β, and the distance to the line x/2 (green) grows. Toggle hard-swish to compare MobileNetV3's piecewise version with SiLU.
f(x) on x ∈ [−5, 5] · drag on the plot to move the probex = 1.00 · f(x) = 0.731 · f′(x) = 0.928−1012345−4−2024x →Swish (β=1)Swish f′ReLUGELU
activation
compare
range [≈ −0.278/β, ∞)monotonic nosmooth yeszero-centred no
β = 1 is SiLU. β = 0 gives x/2; β → ∞ approaches ReLU. Minimum ≈ −0.278/β at x ≈ −1.278/β.
signal through the nonlinearity · histogram of f(x) for x ~ N(0.0, 1.00²)exact quadrature, no samplingE[f]−0.281.823.92output value f(x) → (axis spans the central ±4σ of inputs)
E[f(x)] 0.207Var[f(x)] 0.313E[f′(x)] 0.500f′ = 0 for 0.0% of inputsE[f′]¹⁰ 9.8e−4
E[f′]¹⁰ is a rough heuristic for the gradient scale after 10 such layers: it ignores the weights and assumes every layer sees this same input distribution.
fig.Swish against ReLU and GELU. Set β to 1.702 and the Swish curve almost covers GELU.

At a glance (β = 1, SiLU)

Range
[−0.2785, ∞)[-0.2785,\ \infty); in general [−0.2785/β, ∞)[-0.2785/\beta,\ \infty)
Monotonic
No. Decreasing on (−∞, −1.2785/β)(-\infty,\,-1.2785/\beta)
Smooth
Yes, infinitely differentiable (for finite β\beta); slope in [−0.0998, 1.0998][-0.0998,\ 1.0998]
Zero-centred
No. For x∼N(0,1)x\sim\mathcal N(0,1) the output has mean 0.2070.207 and std 0.5600.560
Saturation
Below only: f→0f\to0 and f′→0f'\to0 as x→−∞x\to-\infty; unbounded above
Cost
One exp and one division per element; hard-swish replaces them with a clamp

Hard-swish

On mobile hardware the exponential is slow and awkward to quantize. MobileNetV3 (Howard et al., 2019) replaced the sigmoid with a piecewise-linear ramp:

h-swish(x)=x⋅ReLU6(x+3)6,ReLU6(t)=min⁡(max⁡(t,0),6)\text{h-swish}(x)=x\cdot\frac{\mathrm{ReLU6}(x+3)}{6},\qquad \mathrm{ReLU6}(t)=\min(\max(t,0),6)
ReLU6(x+3)/6\mathrm{ReLU6}(x+3)/6
a "hard sigmoid": 0 below −3-3, 1 above 33, linear in between

It is exactly 0 for x≤−3x\le-3 and exactly xx for x≥3x\ge3. Its largest deviation from SiLU is 0.14230.1423, reached at x=±3x=\pm3, and its minimum is −0.375-0.375 at x=−1.5x=-1.5, deeper than SiLU's. It is piecewise quadratic, so its slope is continuous but has corners.

Worked example

xσ(x)SiLU(x)SiLU′(x)h-swish(x)
−30.0474−0.1423−0.08810.0000
−20.1192−0.2384−0.0908−0.3333
−1.27850.2178−0.27850.0000−0.3668
−10.2689−0.26890.0723−0.3333
−0.50.3775−0.18880.2600−0.2083
00.50000.00000.50000.0000
10.73110.73110.92770.6667
20.88081.76161.09081.6667
30.95262.85771.08813.0000
SiLU (β = 1), its slope, and hard-swish (4 decimals; highlighted: the minimum)

Check one row by hand. At x=−0.5x=-0.5: σ(−0.5)=0.3775\sigma(-0.5)=0.3775, so f=−0.1888f=-0.1888, and f′=0.3775+(−0.5)(0.3775)(0.6225)=0.3775−0.1175=0.2600f'=0.3775+(-0.5)(0.3775)(0.6225)=0.3775-0.1175=0.2600. The same input through ReLU gives output 0 and slope 0.

Effect on training, and where it is used

Like GELU, Swish passes a small gradient for negative inputs (no dead units), is smooth, and has a ReLU-like slope near 1 for large positive inputs, so deep stacks behave much like ReLU stacks. For unit-variance inputs E[SiLU(z)2]=0.356\mathbb{E}[\mathrm{SiLU}(z)^2]=0.356 against ReLU's 0.50.5, so a plain stack with He initialization starts with slightly shrinking activations. Normalization layers make this irrelevant in most modern architectures.

Where it's used

EfficientNet

Tan & Le (2019) use Swish (β=1\beta=1) throughout the EfficientNet family.

MobileNetV3

Hard-swish mainly in the second half of the network, where the paper found it pays off, with ReLU in most early layers.

LLMs via SwiGLU

LLaMA and PaLM use a feed-forward block W2(SiLU(W1x)⊙W3x)W_2\big(\mathrm{SiLU}(W_1x)\odot W_3x\big) (SwiGLU, Shazeer 2020). There SiLU gates a separate linear projection instead of itself.

YOLO detectors

Later YOLOv5 releases and several successors use nn.SiLU as their default activation.

python
1import torch, torch.nn as nn, torch.nn.functional as F
2
3x = torch.tensor([-2., -1., 0., 1., 2.], requires_grad=True)
4print(nn.SiLU()(x)) # [-0.2384, -0.2689, 0.0000, 0.7311, 1.7616]
5print(F.hardswish(x)) # [-0.3333, -0.3333, 0.0000, 0.6667, 1.6667]
6
7class Swish(nn.Module): # from scratch, β learned (one per layer)
8 def __init__(self, beta=1.0):
9 super().__init__()
10 self.beta = nn.Parameter(torch.tensor(beta))
11 def forward(self, x):
12 return x * torch.sigmoid(self.beta * x)
13
14Swish()(x).sum().backward()
15print(x.grad) # [-0.0908, 0.0723, 0.5000, 0.9277, 1.0908]
SiLU, hard-swish and a Swish with a trainable β

Use when

  • +Drop-in ReLU replacement with no extra parameters (at fixed β).
  • +Smooth, non-zero gradient for negative inputs.
  • +β interpolates between linear and ReLU, and the hard-swish form is cheap enough for mobile.
  • +SiLU is the gate inside SwiGLU, the standard LLM feed-forward block.

Avoid when

  • −An exp and a division per element: slower than ReLU when not fused.
  • −Not monotonic and not zero-centred.
  • −Gains over ReLU are modest (fractions of a percent in the original paper) and do not appear in every setting.

Check yourself

You double β. What happens to the location and depth of Swish's negative minimum?

Check yourself

What is the slope of x·σ(βx) at x = 0?

Remember

  • ✓swishβ(x)=xσ(βx)\mathrm{swish}_\beta(x)=x\sigma(\beta x); with β=1\beta=1 it is SiLU (Hendrycks & Gimpel 2016; Elfwing et al. 2017), and Ramachandran et al. found it by automated search (2017).
  • ✓f′(x)=σ(βx)+βx σ(βx)(1−σ(βx))f'(x)=\sigma(\beta x)+\beta x\,\sigma(\beta x)(1-\sigma(\beta x)), f′(0)=12f'(0)=\tfrac12 for every β\beta.
  • ✓β→0\beta\to0 gives x/2x/2, β→∞\beta\to\infty gives ReLU; the minimum −0.2785/β-0.2785/\beta at x=−1.2785/βx=-1.2785/\beta scales as 1/β1/\beta.
  • ✓Hard-swish x ReLU6(x+3)/6x\,\mathrm{ReLU6}(x+3)/6 (MobileNetV3) is within 0.14230.1423 of SiLU.
  • ✓Used in EfficientNet, MobileNetV3 (hard-swish) and LLaMA/PaLM through SwiGLU.
← back to Deep Learning roadmap