# Mish

> x·tanh(softplus(x)): a smooth, non-monotonic Swish-like curve with a deeper dip, popularised by YOLOv4; its gains over Swish are small and inconsistent.

Source: https://www.mlfoundry.in/learn/mish  
Section: Deep Learning › Activation Functions  
Level: advanced · 8 min read · updated 2026-10-11

**Mish:** is the activation $\mathrm{mish}(x)=x\,\tanh\big(\mathrm{softplus}(x)\big)=x\tanh\big(\ln(1+e^x)\big)$. Like Swish it multiplies the input by a smooth gate that rises from 0 to 1, but the gate is $\tanh(\mathrm{softplus}(x))$ instead of a sigmoid.

Diganta Misra introduced Mish in 2019 ("Mish: A Self Regularized Non-Monotonic Activation Function"), building on Swish. The goal was the same shape family (unbounded above, bounded below, smooth, a small negative dip) with a slightly different gate. The paper reports small accuracy gains over Swish and ReLU on several image benchmarks.

Its best-known use is YOLOv4 (Bochkovskiy et al., 2020), which uses Mish in its CSPDarknet53 backbone. Few other major architectures adopted it. Later YOLO versions moved to SiLU, and transformers use GELU or SwiGLU.

## Formula and derivative

$$
\mathrm{mish}(x)=x\tanh\big(\mathrm{softplus}(x)\big)=x\cdot\frac{e^{2x}+2e^{x}}{e^{2x}+2e^{x}+2}
$$

- $\mathrm{softplus}(x)$: $\ln(1+e^x)$, a smooth ReLU (always positive)
- $\tanh(\mathrm{softplus}(x))$: the gate, in $(0,1)$; with $u=1+e^x$ it equals $(u^2-1)/(u^2+1)$, which gives the rational form on the right

Differentiate the product. The gate's own derivative is $\operatorname{sech}^2(\mathrm{softplus}(x))\cdot\sigma(x)$, because $\mathrm{softplus}'=\sigma$:

$$
\mathrm{mish}'(x)=\tanh\big(\mathrm{sp}(x)\big)+x\,\operatorname{sech}^2\big(\mathrm{sp}(x)\big)\,\sigma(x)=\frac{e^{x}\,\omega}{\delta^{2}}
$$

- $\mathrm{sp}$: softplus
- $\omega$: $4(x+1)+4e^{2x}+e^{3x}+e^{x}(4x+6)$
- $\delta$: $2e^{x}+e^{2x}+2$ (the closed form in the Mish paper; it agrees with the product-rule form to machine precision)

Numbers from solving $\mathrm{mish}'(x)=0$: the minimum is $-0.3088$ at $x=-1.1924$, deeper than SiLU's $-0.2785$ and GELU's $-0.1700$. The slope ranges over $[-0.1125,\ 1.0885]$, with the peak at $x=1.4906$. At zero, $\mathrm{mish}(0)=0$, $\mathrm{mish}'(0)=\tanh(\ln2)=0.6$ and $\mathrm{mish}''(0)=0.64$. Unlike GELU and Swish, Mish does **not** satisfy $f(x)-f(-x)=x$, so its slopes at $\pm x$ do not sum to 1: for $x\sim\mathcal N(0,1)$ the mean slope is $0.564$, not $0.5$.

**At a glance**

- **Range:** $[-0.3088,\ \infty)$; bounded below, unbounded above
- **Monotonic:** No. Decreasing on $(-\infty,\,-1.1924)$
- **Smooth:** Yes, infinitely differentiable
- **Zero-centred:** No. For $x\sim\mathcal N(0,1)$ the output has mean $0.240$ and std $0.628$
- **Saturation:** Below only; above, $\mathrm{mish}(x)\to x$ quickly ($\mathrm{mish}(3)=x-0.0135$)
- **Cost:** exp, log1p and tanh in the naive form (one exp in the rational form); more than Swish's exp and divide

## Mish next to Swish and GELU

> Figure: Mish against Swish (β = 1), GELU and ReLU. The three smooth curves almost coincide for x > 2 and differ mainly in how deep and how wide the negative dip is.

Mish and its neighbours (4 decimals; highlighted: Mish's minimum)

| x | Mish(x) | Mish′(x) | SiLU(x) | GELU(x) |
| --- | --- | --- | --- | --- |
| −3 | −0.1456 | −0.0934 | −0.1423 | −0.0040 |
| −2 | −0.2525 | −0.1084 | −0.2384 | −0.0455 |
| −1.1924 | −0.3088 | 0.0000 | −0.2776 | −0.1390 |
| −1 | −0.3034 | 0.0592 | −0.2689 | −0.1587 |
| −0.5 | −0.2207 | 0.2895 | −0.1888 | −0.1543 |
| 0 | 0.0000 | 0.6000 | 0.0000 | 0.0000 |
| 0.5 | 0.3752 | 0.8864 | 0.3112 | 0.3457 |
| 1 | 0.8651 | 1.0490 | 0.7311 | 0.8413 |
| 2 | 1.9440 | 1.0693 | 1.7616 | 1.9545 |

For $0<x\lesssim1$ Mish is above both SiLU and GELU, because it starts with slope $0.6$ instead of $0.5$; by $x=2$ it sits between them. For $x<0$ it keeps the most negative mass: at $x=-2$ GELU is almost 0 ($-0.0455$), while Mish still outputs $-0.2525$. A unit in that region gets slope $-0.108$ from Mish, $-0.091$ from SiLU and $-0.085$ from GELU.

## Smooth versus piecewise-linear landscapes

The real difference between the smooth family and ReLU appears in whole networks. A ReLU network computes a **piecewise-linear** function of its input: the input space splits into polygons, and the output is exactly affine in each one. Its second derivative is zero almost everywhere and infinite on the creases. Swish, Mish and GELU networks are smooth everywhere, with curvature spread across the input space.

> Figure: Top: f, f′ and f″ for ReLU, Swish, Mish and GELU. ReLU's f′ jumps and its f″ is a spike at 0; the others have bell-shaped curvature, highest for GELU at 0 (0.798) and 0.64 for Mish. Bottom: one random 2-8-8-1 network with identical weights, evaluated with each activation. ReLU's contours are polylines with corners, and its output is exactly affine on a large share of the grid. The smooth activations bend everywhere. Change the input range or draw a new network.

In the default network the ReLU output is exactly affine at 38% of the grid points (all second differences vanish), against 0% for the three smooth activations, whose contours differ from each other only slightly. That is the honest summary of Mish: it belongs to the smooth family, and inside that family the members are close.

## Effect on training

Mish has a slope of $0.6$ at the origin and $\mathbb{E}[\mathrm{mish}(z)^2]=0.452$ for unit-variance inputs. Both are a little larger than SiLU's ($0.5$ and $0.356$), so signals shrink a bit less per layer in an un-normalised stack initialised for ReLU. Negative inputs keep a gradient, so there are no dead units. The paper calls Mish "self regularized" and credits its smoothness and non-monotonic dip for the gains. These are plausible hypotheses, not established mechanisms.

```python
import torch, torch.nn as nn, torch.nn.functional as F

x = torch.tensor([-2., -1., 0., 1., 2.], dtype=torch.float64, requires_grad=True)
print(nn.Mish()(x))                     # [-0.2525, -0.3034, 0.0000, 0.8651, 1.9440]

def mish(x):                            # from scratch; F.softplus is the overflow-safe log(1+e^x)
    return x * torch.tanh(F.softplus(x))

mish(x).sum().backward()
print(x.grad)                           # [-0.1084, 0.0592, 0.6000, 1.0490, 1.0693]
print(torch.autograd.gradcheck(mish, (x,)))   # True

backbone = nn.Sequential(nn.Conv2d(3, 32, 3, padding=1), nn.BatchNorm2d(32), nn.Mish())  # YOLOv4-style conv block
```

Module, functional and a from-scratch version (with a gradient check)

> **Do not write exp(x) directly:** $\ln(1+e^x)$ overflows in float32 once $x>88.7$, and the rational form $e^{2x}$ already overflows at $x>44.4$. Use `F.softplus` (or `nn.Mish`), or return $x$ itself for large $x$: $1-\tanh(\mathrm{softplus}(5))$ is already $9\times10^{-5}$.

> **Small and inconsistent gains:** Reported differences between Mish, Swish and GELU are usually a few tenths of a percent, often comparable to the variation between random seeds, and the ranking changes with architecture, normalization and training recipe. Mish is a reasonable choice when reproducing a model that used it (YOLOv4). It is not a general upgrade, and it costs more per element than ReLU or SiLU.

**Strengths**

- Smooth and non-monotonic, with a gradient for negative inputs (no dead units).
- Slope 0.6 at the origin: slightly stronger signal propagation than SiLU in plain stacks.
- Native in PyTorch (nn.Mish) and the activation of YOLOv4's backbone.

**Limitations**

- The most expensive of the common smooth activations (softplus plus tanh), unless implemented in the rational form.
- Gains over Swish/GELU are small and do not reproduce consistently.
- Not zero-centred, and the deepest negative dip of the family (−0.309).

**Question:** What is Mish's slope at x = 0?

- A. 0.5, like Swish and GELU
- B. tanh(ln 2) = 0.6
- C. 1
- D. 0.64

**Answer:** B. tanh(ln 2) = 0.6 — $\mathrm{mish}'(0)=\tanh(\mathrm{softplus}(0))+0=\tanh(\ln2)=\frac{4-1}{4+1}=0.6$. Swish and GELU have $0.5$ there because $f(x)-f(-x)=x$, which Mish does not satisfy. $0.64$ is $\mathrm{mish}''(0)$.

**Question:** Why does a ReLU network's output landscape look like a patchwork of flat facets, while a Mish network's does not?

- A. ReLU networks have fewer parameters
- B. Composing affine maps with a piecewise-linear activation gives a piecewise-linear function; Mish is smooth, so the composition is smooth
- C. Mish networks are always shallower
- D. ReLU's output is bounded

**Answer:** B. Composing affine maps with a piecewise-linear activation gives a piecewise-linear function; Mish is smooth, so the composition is smooth — Every ReLU unit is affine on each side of its kink, so the network is affine on every region where no unit changes side, and its second derivative is 0 there. Mish has non-zero curvature almost everywhere, so the network bends everywhere. The weights can be identical, as in the figure.

**Key points**

- $\mathrm{mish}(x)=x\tanh(\mathrm{softplus}(x))$ (Misra 2019); smooth, non-monotonic, bounded below, unbounded above.
- $\mathrm{mish}'(x)=\tanh(\mathrm{sp}(x))+x\,\operatorname{sech}^2(\mathrm{sp}(x))\,\sigma(x)$; $\mathrm{mish}'(0)=0.6$.
- Minimum $-0.3088$ at $x=-1.1924$, the deepest dip among Mish, SiLU ($-0.2785$) and GELU ($-0.1700$).
- Best known from YOLOv4's backbone; later YOLOs and most other models use SiLU or GELU.
- Costs more than ReLU and SiLU; gains over Swish/GELU are small and setup-dependent.

---

From MLFoundry (https://www.mlfoundry.in). The web page adds interactive figures and diagrams.
