Definition
Mish is the activation . Like Swish it multiplies the input by a smooth gate that rises from 0 to 1, but the gate is instead of a sigmoid.
Diganta Misra introduced Mish in 2019 ("Mish: A Self Regularized Non-Monotonic Activation Function"), building on Swish. The goal was the same shape family (unbounded above, bounded below, smooth, a small negative dip) with a slightly different gate. The paper reports small accuracy gains over Swish and ReLU on several image benchmarks.
Its best-known use is YOLOv4 (Bochkovskiy et al., 2020), which uses Mish in its CSPDarknet53 backbone. Few other major architectures adopted it. Later YOLO versions moved to SiLU, and transformers use GELU or SwiGLU.
Formula and derivative
- , a smooth ReLU (always positive)
- the gate, in ; with it equals , which gives the rational form on the right
Differentiate the product. The gate's own derivative is , because :
- softplus
- (the closed form in the Mish paper; it agrees with the product-rule form to machine precision)
Numbers from solving : the minimum is at , deeper than SiLU's and GELU's . The slope ranges over , with the peak at . At zero, , and . Unlike GELU and Swish, Mish does not satisfy , so its slopes at do not sum to 1: for the mean slope is , not .
At a glance
- Range
- ; bounded below, unbounded above
- Monotonic
- No. Decreasing on
- Smooth
- Yes, infinitely differentiable
- Zero-centred
- No. For the output has mean and std
- Saturation
- Below only; above, quickly ()
- Cost
- exp, log1p and tanh in the naive form (one exp in the rational form); more than Swish's exp and divide
Mish next to Swish and GELU
| x | Mish(x) | Mish′(x) | SiLU(x) | GELU(x) |
|---|---|---|---|---|
| −3 | −0.1456 | −0.0934 | −0.1423 | −0.0040 |
| −2 | −0.2525 | −0.1084 | −0.2384 | −0.0455 |
| −1.1924 | −0.3088 | 0.0000 | −0.2776 | −0.1390 |
| −1 | −0.3034 | 0.0592 | −0.2689 | −0.1587 |
| −0.5 | −0.2207 | 0.2895 | −0.1888 | −0.1543 |
| 0 | 0.0000 | 0.6000 | 0.0000 | 0.0000 |
| 0.5 | 0.3752 | 0.8864 | 0.3112 | 0.3457 |
| 1 | 0.8651 | 1.0490 | 0.7311 | 0.8413 |
| 2 | 1.9440 | 1.0693 | 1.7616 | 1.9545 |
For Mish is above both SiLU and GELU, because it starts with slope instead of ; by it sits between them. For it keeps the most negative mass: at GELU is almost 0 (), while Mish still outputs . A unit in that region gets slope from Mish, from SiLU and from GELU.
Smooth versus piecewise-linear landscapes
The real difference between the smooth family and ReLU appears in whole networks. A ReLU network computes a piecewise-linear function of its input: the input space splits into polygons, and the output is exactly affine in each one. Its second derivative is zero almost everywhere and infinite on the creases. Swish, Mish and GELU networks are smooth everywhere, with curvature spread across the input space.
In the default network the ReLU output is exactly affine at 38% of the grid points (all second differences vanish), against 0% for the three smooth activations, whose contours differ from each other only slightly. That is the honest summary of Mish: it belongs to the smooth family, and inside that family the members are close.
Effect on training
Mish has a slope of at the origin and for unit-variance inputs. Both are a little larger than SiLU's ( and ), so signals shrink a bit less per layer in an un-normalised stack initialised for ReLU. Negative inputs keep a gradient, so there are no dead units. The paper calls Mish "self regularized" and credits its smoothness and non-monotonic dip for the gains. These are plausible hypotheses, not established mechanisms.
1import torch, torch.nn as nn, torch.nn.functional as F23x = torch.tensor([-2., -1., 0., 1., 2.], dtype=torch.float64, requires_grad=True)4print(nn.Mish()(x)) # [-0.2525, -0.3034, 0.0000, 0.8651, 1.9440]56def mish(x): # from scratch; F.softplus is the overflow-safe log(1+e^x)7 return x * torch.tanh(F.softplus(x))89mish(x).sum().backward()10print(x.grad) # [-0.1084, 0.0592, 0.6000, 1.0490, 1.0693]11print(torch.autograd.gradcheck(mish, (x,))) # True1213backbone = nn.Sequential(nn.Conv2d(3, 32, 3, padding=1), nn.BatchNorm2d(32), nn.Mish()) # YOLOv4-style conv blockUse when
- +Smooth and non-monotonic, with a gradient for negative inputs (no dead units).
- +Slope 0.6 at the origin: slightly stronger signal propagation than SiLU in plain stacks.
- +Native in PyTorch (nn.Mish) and the activation of YOLOv4's backbone.
Avoid when
- −The most expensive of the common smooth activations (softplus plus tanh), unless implemented in the rational form.
- −Gains over Swish/GELU are small and do not reproduce consistently.
- −Not zero-centred, and the deepest negative dip of the family (−0.309).
Check yourself
What is Mish's slope at x = 0?
Check yourself
Why does a ReLU network's output landscape look like a patchwork of flat facets, while a Mish network's does not?
Remember
- ✓ (Misra 2019); smooth, non-monotonic, bounded below, unbounded above.
- ✓; .
- ✓Minimum at , the deepest dip among Mish, SiLU () and GELU ().
- ✓Best known from YOLOv4's backbone; later YOLOs and most other models use SiLU or GELU.
- ✓Costs more than ReLU and SiLU; gains over Swish/GELU are small and setup-dependent.