Mish

x·tanh(softplus(x)): a smooth, non-monotonic Swish-like curve with a deeper dip, popularised by YOLOv4; its gains over Swish are small and inconsistent.

8 min read589 wordsadvanced0 viewsupdated 2026-10-11

Definition

Mish is the activation mish(x)=x tanh⁡(softplus(x))=xtanh⁡(ln⁡(1+ex))\mathrm{mish}(x)=x\,\tanh\big(\mathrm{softplus}(x)\big)=x\tanh\big(\ln(1+e^x)\big). Like Swish it multiplies the input by a smooth gate that rises from 0 to 1, but the gate is tanh⁡(softplus(x))\tanh(\mathrm{softplus}(x)) instead of a sigmoid.

Diganta Misra introduced Mish in 2019 ("Mish: A Self Regularized Non-Monotonic Activation Function"), building on Swish. The goal was the same shape family (unbounded above, bounded below, smooth, a small negative dip) with a slightly different gate. The paper reports small accuracy gains over Swish and ReLU on several image benchmarks.

Its best-known use is YOLOv4 (Bochkovskiy et al., 2020), which uses Mish in its CSPDarknet53 backbone. Few other major architectures adopted it. Later YOLO versions moved to SiLU, and transformers use GELU or SwiGLU.

Formula and derivative

mish(x)=xtanh⁡(softplus(x))=x⋅e2x+2exe2x+2ex+2\mathrm{mish}(x)=x\tanh\big(\mathrm{softplus}(x)\big)=x\cdot\frac{e^{2x}+2e^{x}}{e^{2x}+2e^{x}+2}
softplus(x)\mathrm{softplus}(x)
ln⁡(1+ex)\ln(1+e^x), a smooth ReLU (always positive)
tanh⁡(softplus(x))\tanh(\mathrm{softplus}(x))
the gate, in (0,1)(0,1); with u=1+exu=1+e^x it equals (u2−1)/(u2+1)(u^2-1)/(u^2+1), which gives the rational form on the right

Differentiate the product. The gate's own derivative is sech⁡2(softplus(x))⋅σ(x)\operatorname{sech}^2(\mathrm{softplus}(x))\cdot\sigma(x), because softplus′=σ\mathrm{softplus}'=\sigma:

mish′(x)=tanh⁡(sp(x))+x sech⁡2(sp(x)) σ(x)=ex ωδ2\mathrm{mish}'(x)=\tanh\big(\mathrm{sp}(x)\big)+x\,\operatorname{sech}^2\big(\mathrm{sp}(x)\big)\,\sigma(x)=\frac{e^{x}\,\omega}{\delta^{2}}
sp\mathrm{sp}
softplus
ω\omega
4(x+1)+4e2x+e3x+ex(4x+6)4(x+1)+4e^{2x}+e^{3x}+e^{x}(4x+6)
δ\delta
2ex+e2x+22e^{x}+e^{2x}+2 (the closed form in the Mish paper; it agrees with the product-rule form to machine precision)

Numbers from solving mish′(x)=0\mathrm{mish}'(x)=0: the minimum is −0.3088-0.3088 at x=−1.1924x=-1.1924, deeper than SiLU's −0.2785-0.2785 and GELU's −0.1700-0.1700. The slope ranges over [−0.1125, 1.0885][-0.1125,\ 1.0885], with the peak at x=1.4906x=1.4906. At zero, mish(0)=0\mathrm{mish}(0)=0, mish′(0)=tanh⁡(ln⁡2)=0.6\mathrm{mish}'(0)=\tanh(\ln2)=0.6 and mish′′(0)=0.64\mathrm{mish}''(0)=0.64. Unlike GELU and Swish, Mish does not satisfy f(x)−f(−x)=xf(x)-f(-x)=x, so its slopes at ±x\pm x do not sum to 1: for x∼N(0,1)x\sim\mathcal N(0,1) the mean slope is 0.5640.564, not 0.50.5.

At a glance

Range
[−0.3088, ∞)[-0.3088,\ \infty); bounded below, unbounded above
Monotonic
No. Decreasing on (−∞, −1.1924)(-\infty,\,-1.1924)
Smooth
Yes, infinitely differentiable
Zero-centred
No. For x∼N(0,1)x\sim\mathcal N(0,1) the output has mean 0.2400.240 and std 0.6280.628
Saturation
Below only; above, mish(x)→x\mathrm{mish}(x)\to x quickly (mish(3)=x−0.0135\mathrm{mish}(3)=x-0.0135)
Cost
exp, log1p and tanh in the naive form (one exp in the rational form); more than Swish's exp and divide

Mish next to Swish and GELU

f(x) on x ∈ [−5, 5] · drag on the plot to move the probex = 1.00 · f(x) = 0.865 · f′(x) = 1.049−1012345−4−2024x →MishMish f′SwishGELUReLU
activation
compare
range [≈ −0.309, ∞)monotonic nosmooth yeszero-centred no
Minimum ≈ −0.309 at x ≈ −1.19. Uses a numerically stable softplus.
signal through the nonlinearity · histogram of f(x) for x ~ N(0.0, 1.00²)exact quadrature, no sampling↑ 32%E[f]−0.311.843.99output value f(x) → (axis spans the central ±4σ of inputs)
E[f(x)] 0.240Var[f(x)] 0.395E[f′(x)] 0.564f′ = 0 for 0.0% of inputsE[f′]¹⁰ 3.2e−3
E[f′]¹⁰ is a rough heuristic for the gradient scale after 10 such layers: it ignores the weights and assumes every layer sees this same input distribution.
fig.Mish against Swish (β = 1), GELU and ReLU. The three smooth curves almost coincide for x > 2 and differ mainly in how deep and how wide the negative dip is.
xMish(x)Mish′(x)SiLU(x)GELU(x)
−3−0.1456−0.0934−0.1423−0.0040
−2−0.2525−0.1084−0.2384−0.0455
−1.1924−0.30880.0000−0.2776−0.1390
−1−0.30340.0592−0.2689−0.1587
−0.5−0.22070.2895−0.1888−0.1543
00.00000.60000.00000.0000
0.50.37520.88640.31120.3457
10.86511.04900.73110.8413
21.94401.06931.76161.9545
Mish and its neighbours (4 decimals; highlighted: Mish's minimum)

For 0<x≲10<x\lesssim1 Mish is above both SiLU and GELU, because it starts with slope 0.60.6 instead of 0.50.5; by x=2x=2 it sits between them. For x<0x<0 it keeps the most negative mass: at x=−2x=-2 GELU is almost 0 (−0.0455-0.0455), while Mish still outputs −0.2525-0.2525. A unit in that region gets slope −0.108-0.108 from Mish, −0.091-0.091 from SiLU and −0.085-0.085 from GELU.

Smooth versus piecewise-linear landscapes

The real difference between the smooth family and ReLU appears in whole networks. A ReLU network computes a piecewise-linear function of its input: the input space splits into polygons, and the output is exactly affine in each one. Its second derivative is zero almost everywhere and infinite on the creases. Swish, Mish and GELU networks are smooth everywhere, with curvature spread across the input space.

f(x)012-4-202f′(x)00.51-4-202f″(x)00.40.8-4-202ReLU: δ(x)ReLUaffine at 38% of pointsrange -0.11 … 1.79Swishaffine at 0% of pointsrange -0.10 … 1.63Mishaffine at 0% of pointsrange -0.14 … 1.73GELUaffine at 0% of pointsrange -0.10 … 1.80one 2-8-8-1 net, same weights, on [−3, 3]²; ■ > 0 ■ < 0, lines = 9 contours
ReLU f″(0) = δSwish f″(0) = 0.500Mish f″(0) = 0.640GELU f″(0) = 0.798
"affine" = all second differences of the output vanish on the 3×3 stencil, so the network is exactly linear there. Only ReLU produces such patches; the smooth activations bend everywhere.
fig.Top: f, f′ and f″ for ReLU, Swish, Mish and GELU. ReLU's f′ jumps and its f″ is a spike at 0; the others have bell-shaped curvature, highest for GELU at 0 (0.798) and 0.64 for Mish. Bottom: one random 2-8-8-1 network with identical weights, evaluated with each activation. ReLU's contours are polylines with corners, and its output is exactly affine on a large share of the grid. The smooth activations bend everywhere. Change the input range or draw a new network.

In the default network the ReLU output is exactly affine at 38% of the grid points (all second differences vanish), against 0% for the three smooth activations, whose contours differ from each other only slightly. That is the honest summary of Mish: it belongs to the smooth family, and inside that family the members are close.

Effect on training

Mish has a slope of 0.60.6 at the origin and E[mish(z)2]=0.452\mathbb{E}[\mathrm{mish}(z)^2]=0.452 for unit-variance inputs. Both are a little larger than SiLU's (0.50.5 and 0.3560.356), so signals shrink a bit less per layer in an un-normalised stack initialised for ReLU. Negative inputs keep a gradient, so there are no dead units. The paper calls Mish "self regularized" and credits its smoothness and non-monotonic dip for the gains. These are plausible hypotheses, not established mechanisms.

python
1import torch, torch.nn as nn, torch.nn.functional as F
2
3x = torch.tensor([-2., -1., 0., 1., 2.], dtype=torch.float64, requires_grad=True)
4print(nn.Mish()(x)) # [-0.2525, -0.3034, 0.0000, 0.8651, 1.9440]
5
6def mish(x): # from scratch; F.softplus is the overflow-safe log(1+e^x)
7 return x * torch.tanh(F.softplus(x))
8
9mish(x).sum().backward()
10print(x.grad) # [-0.1084, 0.0592, 0.6000, 1.0490, 1.0693]
11print(torch.autograd.gradcheck(mish, (x,))) # True
12
13backbone = nn.Sequential(nn.Conv2d(3, 32, 3, padding=1), nn.BatchNorm2d(32), nn.Mish()) # YOLOv4-style conv block
Module, functional and a from-scratch version (with a gradient check)

Use when

  • +Smooth and non-monotonic, with a gradient for negative inputs (no dead units).
  • +Slope 0.6 at the origin: slightly stronger signal propagation than SiLU in plain stacks.
  • +Native in PyTorch (nn.Mish) and the activation of YOLOv4's backbone.

Avoid when

  • −The most expensive of the common smooth activations (softplus plus tanh), unless implemented in the rational form.
  • −Gains over Swish/GELU are small and do not reproduce consistently.
  • −Not zero-centred, and the deepest negative dip of the family (−0.309).

Check yourself

What is Mish's slope at x = 0?

Check yourself

Why does a ReLU network's output landscape look like a patchwork of flat facets, while a Mish network's does not?

Remember

  • ✓mish(x)=xtanh⁡(softplus(x))\mathrm{mish}(x)=x\tanh(\mathrm{softplus}(x)) (Misra 2019); smooth, non-monotonic, bounded below, unbounded above.
  • ✓mish′(x)=tanh⁡(sp(x))+x sech⁡2(sp(x)) σ(x)\mathrm{mish}'(x)=\tanh(\mathrm{sp}(x))+x\,\operatorname{sech}^2(\mathrm{sp}(x))\,\sigma(x); mish′(0)=0.6\mathrm{mish}'(0)=0.6.
  • ✓Minimum −0.3088-0.3088 at x=−1.1924x=-1.1924, the deepest dip among Mish, SiLU (−0.2785-0.2785) and GELU (−0.1700-0.1700).
  • ✓Best known from YOLOv4's backbone; later YOLOs and most other models use SiLU or GELU.
  • ✓Costs more than ReLU and SiLU; gains over Swish/GELU are small and setup-dependent.
← back to Deep Learning roadmap