Definition
3D Gaussian splatting (3DGS) 3D Gaussian splatting is an explicit, differentiable scene representation and renderer from Kerbl, Kopanas, Leimkühler and Drettakis, "3D Gaussian Splatting for Real-Time Radiance Field Rendering" (ACM Transactions on Graphics, SIGGRAPH 2023). A scene is a set of 3-D Gaussians. Each has a mean , a covariance , an opacity and view-dependent colour stored as spherical-harmonic coefficients. To render a view, every Gaussian is projected to a 2-D elliptical splat. The splats are sorted by depth inside screen tiles and alpha-composited front to back. All parameters are optimised by gradient descent against posed photographs. Gaussians are added and removed during training (adaptive density control).
Neural radiance fields (NeRF) gave photorealistic novel views but rendered by marching rays through an MLP, 192 network evaluations per pixel. Faster successors (Plenoxels, Instant-NGP) used grids or hash tables but still ray-marched, and the best quality (Mip-NeRF 360, 2022) needed about 48 hours of training per scene and seconds per frame. Kerbl et al. kept the image-formation model of NeRF but changed the primitive. Points-based rendering had long used "splats", oriented discs or ellipses, because GPUs rasterise them quickly (Zwicker et al., EWA splatting, 2001). A 3-D Gaussian is a splat with three properties a learner needs. It is differentiable everywhere. It projects in closed form to a 2-D Gaussian. It can be stretched into a needle or flattened into a disc to fit thin structures and surfaces. The result matched or beat Mip-NeRF 360 in quality, trained in tens of minutes, and rendered at over 100 frames per second.
The primitive
- centre; initialised from the sparse point cloud of structure-from-motion (COLMAP)
- per-axis scales, stored as and activated with
- rotation from a unit quaternion (normalised after each step)
- positive semi-definite by construction. Optimising directly could leave the set of valid covariances.
- opacity, a sigmoid of a free parameter; multiplies when compositing
- spherical harmonics up to degree 3, so 16 coefficients per colour channel: 48 numbers giving view-dependent RGB
Each Gaussian therefore carries floats. Real scenes use 1 to 5 million of them, hundreds of megabytes per scene, against a few megabytes of MLP weights for NeRF.
Projection to a 2-D splat
For a camera with world-to-camera rotation , the mean is transformed and projected as a point. The covariance is pushed through the local affine approximation of the projection, as in EWA splatting:
- the Gaussian's mean in camera coordinates; is its depth
- Jacobian of perspective projection at the mean (focal length in pixels); the paper writes the 3×3 form and keeps the upper-left 2×2 block of the result
- 2×2 screen-space covariance of the splat; the reference rasteriser adds px² to its diagonal as a low-pass filter
- projected 2-D mean; is a pixel centre
Compositing: the same sum as NeRF
- splats overlapping the pixel, ordered front to back by view-space depth
- colour evaluated from the SH coefficients for the view direction
- transmittance ; NeRF's with is the same product
The two methods share the image-formation model. They differ in where the come from. NeRF samples a continuous field at points along a ray and must find the surfaces by querying everywhere. 3DGS has explicit primitives that already sit on the surfaces. Each one is projected once per frame and reused by every pixel it covers, so nothing is evaluated in empty space.
Tile-based rasteriser
One frame
- 1Cull Gaussians whose 99% confidence region is outside the view frustum or that are too close to the near plane. Project the rest, giving , , and a radius of 3 standard deviations along the major axis.
- 2Split the screen into 16×16-pixel tiles. Instantiate one copy of a Gaussian per tile it overlaps, with a 64-bit key: tile ID in the high bits, view-space depth in the low 32 bits.
- 3Sort all keys with one GPU radix sort. Each tile now holds a contiguous, depth-ordered list. The sort is per splat, not per pixel, so the order within a tile is approximate.
- 4Launch one thread block per tile. The block loads its list into shared memory, and each thread (pixel) accumulates colour and transmittance front to back. Splats with are skipped, is clamped to , and a pixel stops once its accumulated opacity would exceed .
- 5The backward pass traverses the same lists back to front. Intermediate transmittances are recovered from the final one, so there is no cap on how many splats can receive gradient from a pixel.
Optimisation and adaptive density control
Training renders a training view, compares it with the photograph, and steps all parameters with Adam. The loss is with . The SfM points give a good but sparse start. Initial covariances are isotropic, with radius equal to the mean distance to the three nearest points. Gradient descent alone cannot add primitives where geometry is missing, so every 100 iterations the method edits the set:
Adaptive density control (Kerbl et al. 2023, Sec. 5.2)
- Trigger
- average magnitude of the view-space positional gradient above : the Gaussian is being pulled hard, so the region is badly reconstructed
- Clone
- under-reconstruction (a small Gaussian): copy it and move the copy along the positional gradient, so volume and count both grow
- Split
- over-reconstruction (a large Gaussian covering detail): replace it by two Gaussians with scales divided by and means sampled from the original Gaussian as a PDF
- Prune
- remove Gaussians with opacity below (0.005 in the code) and ones that are very large in world or view space
- Opacity reset
- every 3000 iterations set all close to zero; the optimiser raises the useful ones again and pruning removes the rest, which controls floaters near the cameras
- Schedule
- 30K iterations; SH degree raised by one band every 1000 iterations starting from degree 0; a short warm-up at 4× lower resolution
Worked example: one Gaussian, one pixel
A Gaussian with scales rotated 30° about the axis, at camera-space position . Take , focal length px and opacity . The px² filter is omitted. On the optical axis , so .
| quantity | value |
|---|---|
| Σ = R diag(0.25, 0.01, 0.01) Rᵀ | [[0.19, 0.1039, 0], [0.1039, 0.07, 0], [0, 0, 0.01]] |
| Σ′ = JΣJᵀ = 625 · (upper-left 2×2) | [[118.75, 64.95], [64.95, 43.75]] px² |
| eigenvalues of Σ′ (std devs) | 156.25 and 6.25 px² (12.5 px and 2.5 px) |
| 3σ radius | ⌈3 · 12.5⌉ = 38 px |
| conic Σ′⁻¹ | [[0.0448, −0.0665], [−0.0665, 0.1216]] |
| α at offset (0, 0) | 0.8 |
| α at offset (10, 0) | 0.8 · e^(−2.24) = 0.0852 |
| α at offset (0, 5) | 0.8 · e^(−1.52) = 0.1750 |
| α at offset (10, 6), near the major axis | 0.8 · e^(−0.4382) = 0.5162 |
Compositing two splats at offset (10, 6)
- 1The projection scales by , so the 3-D standard deviations and become px and px. The rotation is kept: the splat's major axis points along .
- 2Pixel offset is 10 px from the centre but off the major axis, so is only . is farther away (11.7 px) but nearly on the axis, so . Anisotropy is what lets one Gaussian cover an edge or a thin structure.
- 3Behind it is a second splat with at this pixel. Front: . Back: . Remaining transmittance , and .
- 4With colours and : , plus times the background.
1import torch, torch.nn.functional as F23def quat_to_rot(q):4 # q (N, 4) = (w, x, y, z), normalised here so any 4-vector is a valid rotation5 w, x, y, z = F.normalize(q, dim=-1).unbind(-1)6 return torch.stack([7 1 - 2 * (y * y + z * z), 2 * (x * y - w * z), 2 * (x * z + w * y),8 2 * (x * y + w * z), 1 - 2 * (x * x + z * z), 2 * (y * z - w * x),9 2 * (x * z - w * y), 2 * (y * z + w * x), 1 - 2 * (x * x + y * y)], -1).view(-1, 3, 3)1011def render(mu, log_s, q, logit_o, rgb, Wc, tc, f, H, W):12 # mu (N,3) world means, Wc (3,3) world->camera rotation, tc (3,) translation, f focal in pixels13 R = quat_to_rot(q)14 M = R * log_s.exp()[:, None, :] # R S15 Sigma = M @ M.transpose(1, 2) # (N,3,3) Sigma = R S S^T R^T16 p = mu @ Wc.T + tc # camera-space means (N,3)17 x, y, z = p.unbind(-1)18 J = torch.zeros(len(mu), 2, 3)19 J[:, 0, 0] = f / z; J[:, 0, 2] = -f * x / z**2 # Jacobian of the perspective projection20 J[:, 1, 1] = f / z; J[:, 1, 2] = -f * y / z**221 T = J @ Wc # (N,2,3)22 S2 = T @ Sigma @ T.transpose(1, 2) + 0.3 * torch.eye(2) # Sigma' = J W Sigma W^T J^T (+ low-pass)23 uv = torch.stack([f * x / z + W / 2, f * y / z + H / 2], -1)24 order = z.argsort() # front to back (one global sort; 3DGS sorts per tile)25 ys, xs = torch.meshgrid(torch.arange(H) + 0.5, torch.arange(W) + 0.5, indexing="ij")26 pix = torch.stack([xs, ys], -1).view(-1, 1, 2) # (P,1,2)27 d = pix - uv[order][None] # (P,N,2)28 power = -0.5 * (d[..., None, :] @ torch.linalg.inv(S2[order]) @ d[..., None]).squeeze(-1).squeeze(-1)29 alpha = (torch.sigmoid(logit_o[order]) * power.exp()).clamp(max=0.99) # (P,N)30 T_i = torch.cumprod(torch.cat([torch.ones_like(alpha[:, :1]), 1 - alpha], 1), 1) # (P,N+1)31 w = alpha * T_i[:, :-1] # same weights as NeRF: T_i * alpha_i32 img = w @ torch.sigmoid(rgb[order]) # degree-0 colour only (no SH)33 return img.view(H, W, 3), T_i[:, -1].view(H, W), uv3435def densify(params, grad2d, tau=2e-4, scene_extent=1.0, pct=0.01):36 # adaptive density control: large view-space gradient -> clone (small Gaussians) or split (large ones).37 # 3DGS averages the 2-D gradient over views every 100 iterations and uses tau = 0.0002 in NDC units.38 big_grad = grad2d > tau39 large = params["log_s"].exp().max(-1).values > pct * scene_extent40 clone, split = big_grad & ~large, big_grad & large41 R = quat_to_rot(params["q"][split])42 s = params["log_s"][split].exp()43 new_mu = params["mu"][split][None] + (R @ (torch.randn(2, *s.shape) * s)[..., None]).squeeze(-1) # 2 draws ~ N(mu, Sigma)44 out = {}45 for k, v in params.items():46 keep = v[~split].detach()47 add_split = v[split].detach().repeat(2, *[1] * (v.dim() - 1))48 if k == "mu": add_split = new_mu.reshape(-1, 3).detach()49 if k == "log_s": add_split = add_split - torch.log(torch.tensor(1.6)) # scale / 1.650 out[k] = torch.cat([keep, v[clone].detach(), add_split]).requires_grad_(True)51 return out, int(clone.sum()), int(split.sum())5253torch.manual_seed(0)54N, H, W = 200, 48, 6455params = dict(mu=torch.randn(N, 3) * 0.6, log_s=torch.full((N, 3), -2.5), q=torch.randn(N, 4),56 logit_o=torch.zeros(N), rgb=torch.randn(N, 3))57for v in params.values(): v.requires_grad_(True)58Wc, tc = torch.eye(3), torch.tensor([0.0, 0.0, 4.0]) # camera 4 units in front of the origin59target = torch.rand(H, W, 3)60img, T, uv = render(**params, Wc=Wc, tc=tc, f=40.0, H=H, W=W)61uv.retain_grad()62loss = (img - target).abs().mean() # 3DGS uses 0.8 * L1 + 0.2 * D-SSIM63loss.backward()64print(img.shape, T.shape, uv.grad.shape) # torch.Size([48, 64, 3]) torch.Size([48, 64]) torch.Size([200, 2])65params, n_clone, n_split = densify(params, uv.grad.norm(dim=-1), tau=1e-4)66print(n_clone, n_split, params["mu"].shape[0]) # 0 20 220 with this seed: all are "large", so 20 splits, 200 - 20 + 2*20Results
| method | PSNR ↑ | SSIM ↑ | LPIPS ↓ | train | FPS | memory |
|---|---|---|---|---|---|---|
| Instant-NGP (big) | 25.59 | 0.699 | 0.331 | 7.5 min | 9.43 | 48 MB |
| Mip-NeRF 360 | 27.69 | 0.792 | 0.237 | 48 h | 0.06 | 8.6 MB |
| 3DGS-7K | 25.60 | 0.770 | 0.279 | 6.4 min | 160 | 523 MB |
| 3DGS-30K | 27.21 | 0.815 | 0.214 | 41.6 min | 134 | 734 MB |
The abstract's target is real-time rendering, at least 30 fps at 1080p. The measured frame rates are well above it, and three to four orders of magnitude faster than Mip-NeRF 360 at similar quality: Mip-NeRF 360 keeps a small lead in PSNR, while 3DGS is better in SSIM and LPIPS. Memory is the price, hundreds of megabytes of Gaussians against a few megabytes of MLP. Much follow-up work (quantisation, pruning, SH distillation) compresses 3DGS scenes by an order of magnitude.
Use when
- +Real-time rendering: 134 fps on Mip-NeRF 360 scenes versus 0.06 fps for Mip-NeRF 360, at comparable quality
- +Training in tens of minutes instead of hours or days
- +Explicit primitives: no computation in empty space, easy editing, compositing and export
- +The same compositing equation as NeRF, so the photometric training recipe carries over
Avoid when
- −Large memory: hundreds of MB per scene (59 floats per Gaussian, millions of Gaussians)
- −Approximate depth ordering causes popping; aliasing when zooming far from the training scales
- −Relies on SfM initialisation and densification heuristics; sparse views give floaters
- −Not a smooth continuous field: surfaces and normals are hard to extract without extra regularisation
Check yourself
Why does 3DGS parametrise each covariance as Σ = RSSᵀRᵀ instead of optimising the six entries of Σ directly?
Check yourself
A pixel is covered by three splats front to back with α = 0.5, 0.5, 0.5. What weight does the third splat get?
Remember
- ✓3DGS scene: anisotropic Gaussians with mean , covariance , opacity and degree-3 SH colour (59 floats each); projected as .
- ✓Rendering is NeRF's sum with explicit opacities: , , splats sorted by depth per 16×16 tile.
- ✓Adaptive density control every 100 iterations: clone small and split large () Gaussians with view-space gradient above 0.0002, prune transparent ones, reset opacity every 3000 iterations.
- ✓Initialised from SfM points; trained with 0.8·L1 + 0.2·D-SSIM for 30K iterations.
- ✓Mip-NeRF 360 scenes: 27.21 PSNR at 134 fps after about 41 min of training, against 27.69 PSNR at 0.06 fps after 48 h for Mip-NeRF 360.