# Swish

> x·σ(βx): the input gates itself through a sigmoid. One knob β slides it from the line x/2 to ReLU; at β = 1 it is the SiLU used in EfficientNet and LLaMA.

Source: https://www.mlfoundry.in/learn/swish  
Section: Deep Learning › Activation Functions  
Level: core · 8 min read · updated 2026-10-11

**Swish:** is the activation $\mathrm{swish}_\beta(x)=x\,\sigma(\beta x)$, the input multiplied by a sigmoid of itself. $\beta$ is a fixed constant or a learned parameter. With $\beta=1$ it is called **SiLU** (sigmoid-weighted linear unit), $x\,\sigma(x)$.

Swish is **self-gated**. An LSTM or GLU gate multiplies a signal by a sigmoid of a *separate* projection. Swish uses the same scalar for both, so it needs no extra parameters and drops in wherever a ReLU was.

It is an unusual case of an activation found by search. Ramachandran, Zoph & Le (2017, "Searching for Activation Functions") used reinforcement learning to search over compositions of simple unary and binary functions, and $x\,\sigma(\beta x)$ was the best they found. Their abstract reports that replacing ReLU with Swish raised ImageNet top-1 accuracy by 0.9% for Mobile NASNet-A and 0.6% for Inception-ResNet-v2.

The $\beta=1$ function was already known. Hendrycks & Gimpel (2016) named it SiLU as a cheaper relative of GELU, and Elfwing, Uchibe & Doya (2017) used it under the same name for reinforcement learning. PyTorch calls it `nn.SiLU`.

## Formula and derivative

$$
\mathrm{swish}_\beta(x)=x\,\sigma(\beta x)=\frac{x}{1+e^{-\beta x}}
$$

- $\sigma$: logistic sigmoid, $\sigma(t)=1/(1+e^{-t})$
- $\beta$: gate sharpness; $\beta=1$ is SiLU, and $\beta$ may be trained per layer or per channel

$$
\mathrm{swish}_\beta'(x)=\sigma(\beta x)+\beta x\,\sigma(\beta x)\big(1-\sigma(\beta x)\big)=\beta f(x)+\sigma(\beta x)\big(1-\beta f(x)\big)
$$

- $\sigma(\beta x)$: the gate value: what the slope would be if the gate were frozen
- $\beta x\,\sigma(1-\sigma)$: change of the gate with x; negative for x < 0, which creates the dip
- $f(x)$: $\mathrm{swish}_\beta(x)$ itself: the second form reuses the forward output, which is how fused kernels compute the backward pass

At $x=0$ the slope is $\sigma(0)=\tfrac12$ for **every** $\beta$. As for GELU, $f(x)-f(-x)=x$, so $f'(x)+f'(-x)=1$.

## One knob from linear to ReLU

Substitute $u=\beta x$: $\mathrm{swish}_\beta(x)=\mathrm{SiLU}(\beta x)/\beta$. Every member of the family is the SiLU curve shrunk by $1/\beta$ along both axes. Two limits follow:

$$
\beta\to0:\ \ \mathrm{swish}_\beta(x)\to\frac{x}{2},\qquad \beta\to\infty:\ \ \mathrm{swish}_\beta(x)\to\max(0,x)
$$

- $x/2$: $\sigma(0)=\tfrac12$ for every input: a linear function, no nonlinearity at all
- $\max(0,x)$: the gate becomes the step $\mathbb{1}[x>0]$: ReLU

The shrinking also controls the dip. SiLU's minimum is at $u^*=-1.2785$, the root of $1+u(1-\sigma(u))=0$, with value $-0.2785$. So $\mathrm{swish}_\beta$ bottoms out at $x=-1.2785/\beta$ with depth $-0.2785/\beta$. That depth is also its largest distance from ReLU anywhere on the real line. The slope ranges from $-0.0998$ (at $x=-2.3994/\beta$) to $1.0998$ (at $x=+2.3994/\beta$), whatever $\beta$ is.

> Figure: Drag β. Faint curves are fixed members of the family. The dot tracks the negative minimum, which moves toward 0 and gets shallower as 1/β. On the right, the distance to ReLU (rose) falls as 0.2785/β, and the distance to the line x/2 (green) grows. Toggle hard-swish to compare MobileNetV3's piecewise version with SiLU.

> Figure: Swish against ReLU and GELU. Set β to 1.702 and the Swish curve almost covers GELU.

**At a glance (β = 1, SiLU)**

- **Range:** $[-0.2785,\ \infty)$; in general $[-0.2785/\beta,\ \infty)$
- **Monotonic:** No. Decreasing on $(-\infty,\,-1.2785/\beta)$
- **Smooth:** Yes, infinitely differentiable (for finite $\beta$); slope in $[-0.0998,\ 1.0998]$
- **Zero-centred:** No. For $x\sim\mathcal N(0,1)$ the output has mean $0.207$ and std $0.560$
- **Saturation:** Below only: $f\to0$ and $f'\to0$ as $x\to-\infty$; unbounded above
- **Cost:** One exp and one division per element; hard-swish replaces them with a clamp

## Hard-swish

On mobile hardware the exponential is slow and awkward to quantize. MobileNetV3 (Howard et al., 2019) replaced the sigmoid with a piecewise-linear ramp:

$$
\text{h-swish}(x)=x\cdot\frac{\mathrm{ReLU6}(x+3)}{6},\qquad \mathrm{ReLU6}(t)=\min(\max(t,0),6)
$$

- $\mathrm{ReLU6}(x+3)/6$: a "hard sigmoid": 0 below $-3$, 1 above $3$, linear in between

It is exactly 0 for $x\le-3$ and exactly $x$ for $x\ge3$. Its largest deviation from SiLU is $0.1423$, reached at $x=\pm3$, and its minimum is $-0.375$ at $x=-1.5$, deeper than SiLU's. It is piecewise quadratic, so its slope is continuous but has corners.

## Worked example

SiLU (β = 1), its slope, and hard-swish (4 decimals; highlighted: the minimum)

| x | σ(x) | SiLU(x) | SiLU′(x) | h-swish(x) |
| --- | --- | --- | --- | --- |
| −3 | 0.0474 | −0.1423 | −0.0881 | 0.0000 |
| −2 | 0.1192 | −0.2384 | −0.0908 | −0.3333 |
| −1.2785 | 0.2178 | −0.2785 | 0.0000 | −0.3668 |
| −1 | 0.2689 | −0.2689 | 0.0723 | −0.3333 |
| −0.5 | 0.3775 | −0.1888 | 0.2600 | −0.2083 |
| 0 | 0.5000 | 0.0000 | 0.5000 | 0.0000 |
| 1 | 0.7311 | 0.7311 | 0.9277 | 0.6667 |
| 2 | 0.8808 | 1.7616 | 1.0908 | 1.6667 |
| 3 | 0.9526 | 2.8577 | 1.0881 | 3.0000 |

Check one row by hand. At $x=-0.5$: $\sigma(-0.5)=0.3775$, so $f=-0.1888$, and $f'=0.3775+(-0.5)(0.3775)(0.6225)=0.3775-0.1175=0.2600$. The same input through ReLU gives output 0 and slope 0.

## Effect on training, and where it is used

Like GELU, Swish passes a small gradient for negative inputs (no dead units), is smooth, and has a ReLU-like slope near 1 for large positive inputs, so deep stacks behave much like ReLU stacks. For unit-variance inputs $\mathbb{E}[\mathrm{SiLU}(z)^2]=0.356$ against ReLU's $0.5$, so a plain stack with He initialization starts with slightly shrinking activations. Normalization layers make this irrelevant in most modern architectures.

- **EfficientNet:** Tan & Le (2019) use Swish ($\beta=1$) throughout the EfficientNet family.
- **MobileNetV3:** Hard-swish mainly in the second half of the network, where the paper found it pays off, with ReLU in most early layers.
- **LLMs via SwiGLU:** LLaMA and PaLM use a feed-forward block $W_2\big(\mathrm{SiLU}(W_1x)\odot W_3x\big)$ (SwiGLU, Shazeer 2020). There SiLU gates a separate linear projection instead of itself.
- **YOLO detectors:** Later YOLOv5 releases and several successors use `nn.SiLU` as their default activation.

```python
import torch, torch.nn as nn, torch.nn.functional as F

x = torch.tensor([-2., -1., 0., 1., 2.], requires_grad=True)
print(nn.SiLU()(x))        # [-0.2384, -0.2689, 0.0000, 0.7311, 1.7616]
print(F.hardswish(x))      # [-0.3333, -0.3333, 0.0000, 0.6667, 1.6667]

class Swish(nn.Module):    # from scratch, β learned (one per layer)
    def __init__(self, beta=1.0):
        super().__init__()
        self.beta = nn.Parameter(torch.tensor(beta))
    def forward(self, x):
        return x * torch.sigmoid(self.beta * x)

Swish()(x).sum().backward()
print(x.grad)              # [-0.0908, 0.0723, 0.5000, 0.9277, 1.0908]
```

SiLU, hard-swish and a Swish with a trainable β

> **A trainable β can drift:** Nothing stops a learned $\beta$ from going to 0 (the unit becomes linear, $x/2$) or negative (the gate opens for *negative* inputs). If you train $\beta$, log it. The Swish paper tried both a fixed and a trainable $\beta$; most later models simply fix $\beta=1$.

> **Why the dip is harmless:** The non-monotonic region contains only the depth $0.2785/\beta$ and slopes no lower than $-0.0998$. It is too small to destabilise training. Its practical effect is that negative inputs keep a small, informative gradient instead of ReLU's hard zero.

**Strengths**

- Drop-in ReLU replacement with no extra parameters (at fixed β).
- Smooth, non-zero gradient for negative inputs.
- β interpolates between linear and ReLU, and the hard-swish form is cheap enough for mobile.
- SiLU is the gate inside SwiGLU, the standard LLM feed-forward block.

**Limitations**

- An exp and a division per element: slower than ReLU when not fused.
- Not monotonic and not zero-centred.
- Gains over ReLU are modest (fractions of a percent in the original paper) and do not appear in every setting.

**Question:** You double β. What happens to the location and depth of Swish's negative minimum?

- A. Both are halved: the minimum moves to −0.639 with value −0.139
- B. The location halves but the depth stays −0.2785
- C. Both double
- D. The minimum disappears for β > 1

**Answer:** A. Both are halved: the minimum moves to −0.639 with value −0.139 — $\mathrm{swish}_\beta(x)=\mathrm{SiLU}(\beta x)/\beta$ scales both axes by $1/\beta$. The minimum is at $-1.2785/\beta$ with value $-0.2785/\beta$, so $\beta=2$ gives $-0.639$ and $-0.139$. It never disappears; it only shrinks toward ReLU's corner.

**Question:** What is the slope of x·σ(βx) at x = 0?

- A. β/4
- B. 1/2, for every β
- C. 1
- D. 0

**Answer:** B. 1/2, for every β — $f'(0)=\sigma(0)+0=\tfrac12$. The $\beta x\,\sigma(1-\sigma)$ term vanishes at $x=0$. $\beta/4$ is the slope of the gate $\sigma(\beta x)$ itself, not of the activation.

**Key points**

- $\mathrm{swish}_\beta(x)=x\sigma(\beta x)$; with $\beta=1$ it is SiLU (Hendrycks & Gimpel 2016; Elfwing et al. 2017), and Ramachandran et al. found it by automated search (2017).
- $f'(x)=\sigma(\beta x)+\beta x\,\sigma(\beta x)(1-\sigma(\beta x))$, $f'(0)=\tfrac12$ for every $\beta$.
- $\beta\to0$ gives $x/2$, $\beta\to\infty$ gives ReLU; the minimum $-0.2785/\beta$ at $x=-1.2785/\beta$ scales as $1/\beta$.
- Hard-swish $x\,\mathrm{ReLU6}(x+3)/6$ (MobileNetV3) is within $0.1423$ of SiLU.
- Used in EfficientNet, MobileNetV3 (hard-swish) and LLaMA/PaLM through SwiGLU.

---

From MLFoundry (https://www.mlfoundry.in). The web page adds interactive figures and diagrams.
