# 3D Gaussian splatting

> 3D Gaussian splatting represents a scene as millions of anisotropic 3-D Gaussians (mean, covariance RSSᵀRᵀ, opacity, spherical-harmonic colour). It projects them to 2-D splats, sorts them per tile and alpha-composites them with NeRF's equation; fitted to photos, it renders novel views in real time.

Source: https://www.mlfoundry.in/learn/3d-gaussian-splatting  
Section: Deep Learning › Generative Deep Learning Algorithms  
Level: advanced · 15 min read · updated 2026-10-12

**3D Gaussian splatting (3DGS):** **3D Gaussian splatting** is an explicit, differentiable scene representation and renderer from Kerbl, Kopanas, Leimkühler and Drettakis, "3D Gaussian Splatting for Real-Time Radiance Field Rendering" (ACM Transactions on Graphics, SIGGRAPH 2023). A scene is a set of 3-D Gaussians. Each has a mean $\boldsymbol\mu$, a covariance $\Sigma=RSS^\top R^\top$, an opacity $o$ and view-dependent colour stored as spherical-harmonic coefficients. To render a view, every Gaussian is projected to a 2-D elliptical **splat**. The splats are sorted by depth inside screen tiles and **alpha-composited** front to back. All parameters are optimised by gradient descent against posed photographs. Gaussians are added and removed during training (**adaptive density control**).

**Neural radiance fields (NeRF)** gave photorealistic novel views but rendered by marching rays through an MLP, 192 network evaluations per pixel. Faster successors (Plenoxels, Instant-NGP) used grids or hash tables but still ray-marched, and the best quality (Mip-NeRF 360, 2022) needed about 48 hours of training per scene and seconds per frame. Kerbl et al. kept the image-formation model of NeRF but changed the primitive. Points-based rendering had long used "splats", oriented discs or ellipses, because GPUs rasterise them quickly (Zwicker et al., EWA splatting, 2001). A 3-D Gaussian is a splat with three properties a learner needs. It is differentiable everywhere. It projects in closed form to a 2-D Gaussian. It can be stretched into a needle or flattened into a disc to fit thin structures and surfaces. The result matched or beat Mip-NeRF 360 in quality, trained in tens of minutes, and rendered at over 100 frames per second.

## The primitive

$$
G(\mathbf x)=\exp\!\Big(-\tfrac12(\mathbf x-\boldsymbol\mu)^\top\Sigma^{-1}(\mathbf x-\boldsymbol\mu)\Big),\qquad \Sigma=R\,S\,S^\top R^\top,\qquad S=\mathrm{diag}(s_1,s_2,s_3),\quad R=R(\mathbf q)
$$

- $\boldsymbol\mu\in\mathbb R^3$: centre; initialised from the sparse point cloud of structure-from-motion (COLMAP)
- $S$: per-axis scales, stored as $\log s$ and activated with $\exp$
- $R(\mathbf q)$: rotation from a unit quaternion $\mathbf q$ (normalised after each step)
- $RSS^\top R^\top$: positive semi-definite by construction. Optimising $\Sigma$ directly could leave the set of valid covariances.
- $o\in(0,1)$: opacity, a sigmoid of a free parameter; multiplies $G$ when compositing
- $\text{SH}$: spherical harmonics up to degree 3, so 16 coefficients per colour channel: 48 numbers giving view-dependent RGB

Each Gaussian therefore carries $3+3+4+1+48=59$ floats. Real scenes use 1 to 5 million of them, hundreds of megabytes per scene, against a few megabytes of MLP weights for NeRF.

## Projection to a 2-D splat

For a camera with world-to-camera rotation $W$, the mean is transformed and projected as a point. The covariance is pushed through the **local affine approximation** of the projection, as in EWA splatting:

$$
\Sigma'=J\,W\,\Sigma\,W^\top J^\top,\qquad J=\begin{pmatrix} f/z & 0 & -f x/z^2\\ 0 & f/z & -f y/z^2\end{pmatrix},\qquad \alpha_i(\mathbf p)=o_i\exp\!\Big(-\tfrac12(\mathbf p-\boldsymbol\mu_i')^\top\Sigma_i'^{-1}(\mathbf p-\boldsymbol\mu_i')\Big)
$$

- $(x,y,z)$: the Gaussian's mean in camera coordinates; $z$ is its depth
- $J$: Jacobian of perspective projection at the mean (focal length $f$ in pixels); the paper writes the 3×3 form and keeps the upper-left 2×2 block of the result
- $\Sigma'$: 2×2 screen-space covariance of the splat; the reference rasteriser adds $0.3$ px² to its diagonal as a low-pass filter
- $\boldsymbol\mu'_i$: projected 2-D mean; $\mathbf p$ is a pixel centre

## Compositing: the same sum as NeRF

$$
C(\mathbf p)=\sum_{i\in\mathcal N}\mathbf c_i(\mathbf d)\,\alpha_i\prod_{j<i}(1-\alpha_j)
$$

- $\mathcal N$: splats overlapping the pixel, ordered front to back by view-space depth
- $\mathbf c_i(\mathbf d)$: colour evaluated from the SH coefficients for the view direction
- $\prod_{j<i}(1-\alpha_j)$: transmittance $T_i$; NeRF's $T_i=\exp(-\sum_{j<i}\sigma_j\delta_j)$ with $\alpha_i=1-e^{-\sigma_i\delta_i}$ is the same product

The two methods share the image-formation model. They differ in where the $\alpha_i$ come from. NeRF samples a continuous field at points along a ray and must find the surfaces by querying everywhere. 3DGS has explicit primitives that already sit on the surfaces. Each one is projected once per frame and reused by every pixel it covers, so nothing is evaluated in empty space.

## Tile-based rasteriser

**One frame**

1. Cull Gaussians whose 99% confidence region is outside the view frustum or that are too close to the near plane. Project the rest, giving $\boldsymbol\mu'$, $\Sigma'$, and a radius of 3 standard deviations along the major axis.
2. Split the screen into **16×16-pixel tiles**. Instantiate one copy of a Gaussian per tile it overlaps, with a 64-bit key: tile ID in the high bits, view-space depth in the low 32 bits.
3. Sort all keys with **one GPU radix sort**. Each tile now holds a contiguous, depth-ordered list. The sort is per splat, not per pixel, so the order within a tile is approximate.
4. Launch one thread block per tile. The block loads its list into shared memory, and each thread (pixel) accumulates colour and transmittance front to back. Splats with $\alpha<1/255$ are skipped, $\alpha$ is clamped to $0.99$, and a pixel stops once its accumulated opacity would exceed $0.9999$.
5. The backward pass traverses the same lists back to front. Intermediate transmittances are recovered from the final one, so there is no cap on how many splats can receive gradient from a pixel.

## Optimisation and adaptive density control

Training renders a training view, compares it with the photograph, and steps all parameters with Adam. The loss is $\mathcal L=(1-\lambda)\mathcal L_1+\lambda\mathcal L_{\text{D-SSIM}}$ with $\lambda=0.2$. The SfM points give a good but sparse start. Initial covariances are isotropic, with radius equal to the mean distance to the three nearest points. Gradient descent alone cannot add primitives where geometry is missing, so every 100 iterations the method edits the set:

**Adaptive density control (Kerbl et al. 2023, Sec. 5.2)**

- **Trigger:** average magnitude of the view-space positional gradient above $\tau_{\text{pos}}=0.0002$: the Gaussian is being pulled hard, so the region is badly reconstructed
- **Clone:** under-reconstruction (a small Gaussian): copy it and move the copy along the positional gradient, so volume and count both grow
- **Split:** over-reconstruction (a large Gaussian covering detail): replace it by two Gaussians with scales divided by $\phi=1.6$ and means sampled from the original Gaussian as a PDF
- **Prune:** remove Gaussians with opacity below $\epsilon_\alpha$ (0.005 in the code) and ones that are very large in world or view space
- **Opacity reset:** every 3000 iterations set all $o$ close to zero; the optimiser raises the useful ones again and pruning removes the rest, which controls floaters near the cameras
- **Schedule:** 30K iterations; SH degree raised by one band every 1000 iterations starting from degree 0; a short warm-up at 4× lower resolution

> Figure: Nine anisotropic 3-D Gaussians, each with one colour (degree-0 SH), rendered at 32 × 24 px with 8 × 8 tiles (16 × 16 in the paper) by a camera 4 units from the origin. At the default yaw of 30° all 9 are in view. The tile lists (corner numbers) hold 4 to 8 splats, the frame needs K = 72 (tile, depth) sort keys, and it performs 1697 α-blends, 2.2 per pixel. G3 is a needle with scales (0.55, 0.07, 0.07) rotated by 40°. It sits at depth 4.06 with 2-D mean (13.0, 7.1) and Σ′ = [15.2 11.1; 11.1 9.4] px². Its 3σ radius of 15 px puts it in all 12 tiles. At the marked pixel (14, 9) three splats contribute front to back. G2 has α = 0.127 at T = 1. G3 has α = 0.381 at T = 0.873 (weight 0.332). G0 has α = 0.101 at T = 0.541. The background keeps T = 0.486. Rotate G3 to 90°: Σ′ becomes [0.8 0.7; 0.7 28.6], a vertical needle. Its α at the pixel falls to 0.202 and T rises to 0.626. Press split: G3 becomes two copies with scales / 1.6 = (0.34, 0.04, 0.04) and means drawn from G3 itself. K rises to 75, and the copy G3a has a 10-px radius and touches 9 tiles. Reset, set G3's opacity to 0 and prune: 8 Gaussians, K = 60, 1568 blends. Turn the yaw and the depth order changes. At 150°, G3 is the front splat at the pixel with α = 0.823.

## Worked example: one Gaussian, one pixel

A Gaussian with scales $s=(0.5,0.1,0.1)$ rotated 30° about the $z$ axis, at camera-space position $(0,0,4)$. Take $W=I$, focal length $f=100$ px and opacity $o=0.8$. The $0.3$ px² filter is omitted. On the optical axis $x=y=0$, so $J=\begin{pmatrix}25&0&0\\0&25&0\end{pmatrix}$.

Projection and per-pixel opacity, computed in Python

| quantity | value |
| --- | --- |
| Σ = R diag(0.25, 0.01, 0.01) Rᵀ | [[0.19, 0.1039, 0], [0.1039, 0.07, 0], [0, 0, 0.01]] |
| Σ′ = JΣJᵀ = 625 · (upper-left 2×2) | [[118.75, 64.95], [64.95, 43.75]] px² |
| eigenvalues of Σ′ (std devs) | 156.25 and 6.25 px² (12.5 px and 2.5 px) |
| 3σ radius | ⌈3 · 12.5⌉ = 38 px |
| conic Σ′⁻¹ | [[0.0448, −0.0665], [−0.0665, 0.1216]] |
| α at offset (0, 0) | 0.8 |
| α at offset (10, 0) | 0.8 · e^(−2.24) = 0.0852 |
| α at offset (0, 5) | 0.8 · e^(−1.52) = 0.1750 |
| α at offset (10, 6), near the major axis | 0.8 · e^(−0.4382) = 0.5162 |

**Compositing two splats at offset (10, 6)**

1. The projection scales $\Sigma$ by $(f/z)^2=625$, so the 3-D standard deviations $0.5$ and $0.1$ become $12.5$ px and $2.5$ px. The rotation is kept: the splat's major axis points along $(\cos30°,\sin30°)$.
2. Pixel offset $(10,0)$ is 10 px from the centre but off the major axis, so $\alpha$ is only $0.085$. $(10,6)$ is farther away (11.7 px) but nearly on the axis, so $\alpha=0.516$. Anisotropy is what lets one Gaussian cover an edge or a thin structure.
3. Behind it is a second splat with $\alpha_B=0.6$ at this pixel. Front: $w_A=\alpha_A=0.5162$. Back: $w_B=(1-0.5162)\cdot0.6=0.2903$. Remaining transmittance $T=0.4838\cdot0.4=0.1935$, and $w_A+w_B+T=1$.
4. With colours $\mathbf c_A=(1,0.2,0.2)$ and $\mathbf c_B=(0.2,0.4,1)$: $C=0.5162\,\mathbf c_A+0.2903\,\mathbf c_B=(0.5743,\,0.2194,\,0.3935)$, plus $0.1935$ times the background.

```python
import torch, torch.nn.functional as F

def quat_to_rot(q):
    # q (N, 4) = (w, x, y, z), normalised here so any 4-vector is a valid rotation
    w, x, y, z = F.normalize(q, dim=-1).unbind(-1)
    return torch.stack([
        1 - 2 * (y * y + z * z), 2 * (x * y - w * z), 2 * (x * z + w * y),
        2 * (x * y + w * z), 1 - 2 * (x * x + z * z), 2 * (y * z - w * x),
        2 * (x * z - w * y), 2 * (y * z + w * x), 1 - 2 * (x * x + y * y)], -1).view(-1, 3, 3)

def render(mu, log_s, q, logit_o, rgb, Wc, tc, f, H, W):
    # mu (N,3) world means, Wc (3,3) world->camera rotation, tc (3,) translation, f focal in pixels
    R = quat_to_rot(q)
    M = R * log_s.exp()[:, None, :]                       # R S
    Sigma = M @ M.transpose(1, 2)                         # (N,3,3)  Sigma = R S S^T R^T
    p = mu @ Wc.T + tc                                    # camera-space means (N,3)
    x, y, z = p.unbind(-1)
    J = torch.zeros(len(mu), 2, 3)
    J[:, 0, 0] = f / z; J[:, 0, 2] = -f * x / z**2        # Jacobian of the perspective projection
    J[:, 1, 1] = f / z; J[:, 1, 2] = -f * y / z**2
    T = J @ Wc                                            # (N,2,3)
    S2 = T @ Sigma @ T.transpose(1, 2) + 0.3 * torch.eye(2)   # Sigma' = J W Sigma W^T J^T (+ low-pass)
    uv = torch.stack([f * x / z + W / 2, f * y / z + H / 2], -1)
    order = z.argsort()                                   # front to back (one global sort; 3DGS sorts per tile)
    ys, xs = torch.meshgrid(torch.arange(H) + 0.5, torch.arange(W) + 0.5, indexing="ij")
    pix = torch.stack([xs, ys], -1).view(-1, 1, 2)        # (P,1,2)
    d = pix - uv[order][None]                             # (P,N,2)
    power = -0.5 * (d[..., None, :] @ torch.linalg.inv(S2[order]) @ d[..., None]).squeeze(-1).squeeze(-1)
    alpha = (torch.sigmoid(logit_o[order]) * power.exp()).clamp(max=0.99)     # (P,N)
    T_i = torch.cumprod(torch.cat([torch.ones_like(alpha[:, :1]), 1 - alpha], 1), 1)  # (P,N+1)
    w = alpha * T_i[:, :-1]                               # same weights as NeRF: T_i * alpha_i
    img = w @ torch.sigmoid(rgb[order])                   # degree-0 colour only (no SH)
    return img.view(H, W, 3), T_i[:, -1].view(H, W), uv

def densify(params, grad2d, tau=2e-4, scene_extent=1.0, pct=0.01):
    # adaptive density control: large view-space gradient -> clone (small Gaussians) or split (large ones).
    # 3DGS averages the 2-D gradient over views every 100 iterations and uses tau = 0.0002 in NDC units.
    big_grad = grad2d > tau
    large = params["log_s"].exp().max(-1).values > pct * scene_extent
    clone, split = big_grad & ~large, big_grad & large
    R = quat_to_rot(params["q"][split])
    s = params["log_s"][split].exp()
    new_mu = params["mu"][split][None] + (R @ (torch.randn(2, *s.shape) * s)[..., None]).squeeze(-1)  # 2 draws ~ N(mu, Sigma)
    out = {}
    for k, v in params.items():
        keep = v[~split].detach()
        add_split = v[split].detach().repeat(2, *[1] * (v.dim() - 1))
        if k == "mu": add_split = new_mu.reshape(-1, 3).detach()
        if k == "log_s": add_split = add_split - torch.log(torch.tensor(1.6))  # scale / 1.6
        out[k] = torch.cat([keep, v[clone].detach(), add_split]).requires_grad_(True)
    return out, int(clone.sum()), int(split.sum())

torch.manual_seed(0)
N, H, W = 200, 48, 64
params = dict(mu=torch.randn(N, 3) * 0.6, log_s=torch.full((N, 3), -2.5), q=torch.randn(N, 4),
              logit_o=torch.zeros(N), rgb=torch.randn(N, 3))
for v in params.values(): v.requires_grad_(True)
Wc, tc = torch.eye(3), torch.tensor([0.0, 0.0, 4.0])      # camera 4 units in front of the origin
target = torch.rand(H, W, 3)
img, T, uv = render(**params, Wc=Wc, tc=tc, f=40.0, H=H, W=W)
uv.retain_grad()
loss = (img - target).abs().mean()                     # 3DGS uses 0.8 * L1 + 0.2 * D-SSIM
loss.backward()
print(img.shape, T.shape, uv.grad.shape)               # torch.Size([48, 64, 3]) torch.Size([48, 64]) torch.Size([200, 2])
params, n_clone, n_split = densify(params, uv.grad.norm(dim=-1), tau=1e-4)
print(n_clone, n_split, params["mu"].shape[0])            # 0 20 220 with this seed: all are "large", so 20 splits, 200 - 20 + 2*20
```

Differentiable Gaussian splatting in plain PyTorch (dense, no tiles), plus clone/split densification. Real implementations use a CUDA tile rasteriser.

## Results

Mip-NeRF 360 dataset, Kerbl et al. (2023) Table 1. FPS on an NVIDIA A6000; 3DGS-7K and -30K are the same method stopped at 7K and 30K iterations.

| method | PSNR ↑ | SSIM ↑ | LPIPS ↓ | train | FPS | memory |
| --- | --- | --- | --- | --- | --- | --- |
| Instant-NGP (big) | 25.59 | 0.699 | 0.331 | 7.5 min | 9.43 | 48 MB |
| Mip-NeRF 360 | 27.69 | 0.792 | 0.237 | 48 h | 0.06 | 8.6 MB |
| 3DGS-7K | 25.60 | 0.770 | 0.279 | 6.4 min | 160 | 523 MB |
| 3DGS-30K | 27.21 | 0.815 | 0.214 | 41.6 min | 134 | 734 MB |

The abstract's target is real-time rendering, at least 30 fps at 1080p. The measured frame rates are well above it, and three to four orders of magnitude faster than Mip-NeRF 360 at similar quality: Mip-NeRF 360 keeps a small lead in PSNR, while 3DGS is better in SSIM and LPIPS. Memory is the price, hundreds of megabytes of Gaussians against a few megabytes of MLP. Much follow-up work (quantisation, pruning, SH distillation) compresses 3DGS scenes by an order of magnitude.

> **Per-tile sorting is not per-pixel sorting:** Splats are ordered by the depth of their centres, once per tile. Two large, intersecting Gaussians can be in the wrong order for some pixels, and the order can flip abruptly as the camera moves ("popping"). The paper accepts this because training adapts the Gaussians to it and they become small. Later work sorts per pixel or per ray (StopThePop, 2024) to remove the artefacts. Do not expect exact volumetric integration. The $\alpha$ of a splat is a 2-D footprint, not an integral of density along the ray.

> **Densification hyperparameters decide quality and memory:** The gradient threshold, the split scale and the opacity-reset period trade quality against the number of Gaussians. Too permissive and a scene grows to many millions of Gaussians and runs out of GPU memory. Too strict and large blurry Gaussians stay. Sparse or poorly posed captures also produce floaters and needle-like splats that look right only from the training views. Watch the Gaussian count during training, not just the loss.

> **An explicit, editable output for generative 3-D:** Gaussians are a point cloud with attributes, so they can be moved, deleted, merged and animated directly. Their fast differentiable renderer makes them an attractive target for generators. **DreamGaussian** (Tang et al., ICLR 2024) runs DreamFusion-style score distillation from a 2-D **Diffusion models** prior on Gaussians instead of a NeRF, and produces a textured mesh in minutes. Feed-forward models such as LGM (2024) predict the Gaussians of an object directly from one or a few images, with no per-scene optimisation. Dynamic and 4-D variants attach motion to each Gaussian.

**Strengths**

- Real-time rendering: 134 fps on Mip-NeRF 360 scenes versus 0.06 fps for Mip-NeRF 360, at comparable quality
- Training in tens of minutes instead of hours or days
- Explicit primitives: no computation in empty space, easy editing, compositing and export
- The same compositing equation as NeRF, so the photometric training recipe carries over

**Limitations**

- Large memory: hundreds of MB per scene (59 floats per Gaussian, millions of Gaussians)
- Approximate depth ordering causes popping; aliasing when zooming far from the training scales
- Relies on SfM initialisation and densification heuristics; sparse views give floaters
- Not a smooth continuous field: surfaces and normals are hard to extract without extra regularisation

**Question:** Why does 3DGS parametrise each covariance as Σ = RSSᵀRᵀ instead of optimising the six entries of Σ directly?

- A. Gradient steps on raw entries can make Σ indefinite; RSSᵀRᵀ is positive semi-definite for any rotation and scales
- B. It halves the number of parameters per Gaussian
- C. It makes Σ diagonal, which the rasteriser requires
- D. So that Σ′ = JWΣWᵀJᵀ can be skipped

**Answer:** A. Gradient steps on raw entries can make Σ indefinite; RSSᵀRᵀ is positive semi-definite for any rotation and scales — A covariance must be positive semi-definite. Unconstrained updates of its entries can leave that set, and then the density is not a Gaussian. With $\Sigma=(RS)(RS)^\top$, every quaternion and scale gives a valid covariance. It is still full and anisotropic (7 parameters, versus 6 for a symmetric matrix). The projection to $\Sigma'$ is still needed.

**Question:** A pixel is covered by three splats front to back with α = 0.5, 0.5, 0.5. What weight does the third splat get?

- A. 0.5
- B. 0.25
- C. 0.125
- D. 0.167

**Answer:** C. 0.125 — $w_3=\alpha_3(1-\alpha_1)(1-\alpha_2)=0.5\cdot0.5\cdot0.5=0.125$. The weights $0.5, 0.25, 0.125$ leave transmittance $0.125$ for the background. This is NeRF's $T_i\alpha_i$ with explicit $\alpha$s.

**Key points**

- 3DGS scene: anisotropic Gaussians with mean $\boldsymbol\mu$, covariance $\Sigma=RSS^\top R^\top$, opacity $o$ and degree-3 SH colour (59 floats each); projected as $\Sigma'=JW\Sigma W^\top J^\top$.
- Rendering is NeRF's sum with explicit opacities: $C=\sum_i\mathbf c_i\alpha_i\prod_{j<i}(1-\alpha_j)$, $\alpha_i=o_i\exp(-\tfrac12\mathbf d^\top\Sigma_i'^{-1}\mathbf d)$, splats sorted by depth per 16×16 tile.
- Adaptive density control every 100 iterations: clone small and split large ($s/1.6$) Gaussians with view-space gradient above 0.0002, prune transparent ones, reset opacity every 3000 iterations.
- Initialised from SfM points; trained with 0.8·L1 + 0.2·D-SSIM for 30K iterations.
- Mip-NeRF 360 scenes: 27.21 PSNR at 134 fps after about 41 min of training, against 27.69 PSNR at 0.06 fps after 48 h for Mip-NeRF 360.

---

From MLFoundry (https://www.mlfoundry.in). The web page adds interactive figures and diagrams.
