3D Gaussian splatting

3D Gaussian splatting represents a scene as millions of anisotropic 3-D Gaussians (mean, covariance RSSᵀRᵀ, opacity, spherical-harmonic colour). It projects them to 2-D splats, sorts them per tile and alpha-composites them with NeRF's equation; fitted to photos, it renders novel views in real time.

15 min read1213 wordsadvanced0 viewsupdated 2026-10-12

Definition

3D Gaussian splatting (3DGS) 3D Gaussian splatting is an explicit, differentiable scene representation and renderer from Kerbl, Kopanas, Leimkühler and Drettakis, "3D Gaussian Splatting for Real-Time Radiance Field Rendering" (ACM Transactions on Graphics, SIGGRAPH 2023). A scene is a set of 3-D Gaussians. Each has a mean μ\boldsymbol\mu, a covariance Σ=RSS⊤R⊤\Sigma=RSS^\top R^\top, an opacity oo and view-dependent colour stored as spherical-harmonic coefficients. To render a view, every Gaussian is projected to a 2-D elliptical splat. The splats are sorted by depth inside screen tiles and alpha-composited front to back. All parameters are optimised by gradient descent against posed photographs. Gaussians are added and removed during training (adaptive density control).

Neural radiance fields (NeRF) gave photorealistic novel views but rendered by marching rays through an MLP, 192 network evaluations per pixel. Faster successors (Plenoxels, Instant-NGP) used grids or hash tables but still ray-marched, and the best quality (Mip-NeRF 360, 2022) needed about 48 hours of training per scene and seconds per frame. Kerbl et al. kept the image-formation model of NeRF but changed the primitive. Points-based rendering had long used "splats", oriented discs or ellipses, because GPUs rasterise them quickly (Zwicker et al., EWA splatting, 2001). A 3-D Gaussian is a splat with three properties a learner needs. It is differentiable everywhere. It projects in closed form to a 2-D Gaussian. It can be stretched into a needle or flattened into a disc to fit thin structures and surfaces. The result matched or beat Mip-NeRF 360 in quality, trained in tens of minutes, and rendered at over 100 frames per second.

The primitive

G(x)=exp⁡ ⁣(−12(x−μ)⊤Σ−1(x−μ)),Σ=R S S⊤R⊤,S=diag(s1,s2,s3),R=R(q)G(\mathbf x)=\exp\!\Big(-\tfrac12(\mathbf x-\boldsymbol\mu)^\top\Sigma^{-1}(\mathbf x-\boldsymbol\mu)\Big),\qquad \Sigma=R\,S\,S^\top R^\top,\qquad S=\mathrm{diag}(s_1,s_2,s_3),\quad R=R(\mathbf q)
μ∈R3\boldsymbol\mu\in\mathbb R^3
centre; initialised from the sparse point cloud of structure-from-motion (COLMAP)
SS
per-axis scales, stored as log⁡s\log s and activated with exp⁡\exp
R(q)R(\mathbf q)
rotation from a unit quaternion q\mathbf q (normalised after each step)
RSS⊤R⊤RSS^\top R^\top
positive semi-definite by construction. Optimising Σ\Sigma directly could leave the set of valid covariances.
o∈(0,1)o\in(0,1)
opacity, a sigmoid of a free parameter; multiplies GG when compositing
SH\text{SH}
spherical harmonics up to degree 3, so 16 coefficients per colour channel: 48 numbers giving view-dependent RGB

Each Gaussian therefore carries 3+3+4+1+48=593+3+4+1+48=59 floats. Real scenes use 1 to 5 million of them, hundreds of megabytes per scene, against a few megabytes of MLP weights for NeRF.

Projection to a 2-D splat

For a camera with world-to-camera rotation WW, the mean is transformed and projected as a point. The covariance is pushed through the local affine approximation of the projection, as in EWA splatting:

Σ′=J W Σ W⊤J⊤,J=(f/z0−fx/z20f/z−fy/z2),αi(p)=oiexp⁡ ⁣(−12(p−μi′)⊤Σi′−1(p−μi′))\Sigma'=J\,W\,\Sigma\,W^\top J^\top,\qquad J=\begin{pmatrix} f/z & 0 & -f x/z^2\\ 0 & f/z & -f y/z^2\end{pmatrix},\qquad \alpha_i(\mathbf p)=o_i\exp\!\Big(-\tfrac12(\mathbf p-\boldsymbol\mu_i')^\top\Sigma_i'^{-1}(\mathbf p-\boldsymbol\mu_i')\Big)
(x,y,z)(x,y,z)
the Gaussian's mean in camera coordinates; zz is its depth
JJ
Jacobian of perspective projection at the mean (focal length ff in pixels); the paper writes the 3×3 form and keeps the upper-left 2×2 block of the result
Σ′\Sigma'
2×2 screen-space covariance of the splat; the reference rasteriser adds 0.30.3 px² to its diagonal as a low-pass filter
μi′\boldsymbol\mu'_i
projected 2-D mean; p\mathbf p is a pixel centre

Compositing: the same sum as NeRF

C(p)=∑i∈Nci(d) αi∏j<i(1−αj)C(\mathbf p)=\sum_{i\in\mathcal N}\mathbf c_i(\mathbf d)\,\alpha_i\prod_{j<i}(1-\alpha_j)
N\mathcal N
splats overlapping the pixel, ordered front to back by view-space depth
ci(d)\mathbf c_i(\mathbf d)
colour evaluated from the SH coefficients for the view direction
∏j<i(1−αj)\prod_{j<i}(1-\alpha_j)
transmittance TiT_i; NeRF's Ti=exp⁡(−∑j<iσjδj)T_i=\exp(-\sum_{j<i}\sigma_j\delta_j) with αi=1−e−σiδi\alpha_i=1-e^{-\sigma_i\delta_i} is the same product

The two methods share the image-formation model. They differ in where the αi\alpha_i come from. NeRF samples a continuous field at points along a ray and must find the surfaces by querying everywhere. 3DGS has explicit primitives that already sit on the surfaces. Each one is projected once per frame and reused by every pixel it covers, so nothing is evaluated in empty space.

Tile-based rasteriser

One frame

  1. 1Cull Gaussians whose 99% confidence region is outside the view frustum or that are too close to the near plane. Project the rest, giving μ′\boldsymbol\mu', Σ′\Sigma', and a radius of 3 standard deviations along the major axis.
  2. 2Split the screen into 16×16-pixel tiles. Instantiate one copy of a Gaussian per tile it overlaps, with a 64-bit key: tile ID in the high bits, view-space depth in the low 32 bits.
  3. 3Sort all keys with one GPU radix sort. Each tile now holds a contiguous, depth-ordered list. The sort is per splat, not per pixel, so the order within a tile is approximate.
  4. 4Launch one thread block per tile. The block loads its list into shared memory, and each thread (pixel) accumulates colour and transmittance front to back. Splats with α<1/255\alpha<1/255 are skipped, α\alpha is clamped to 0.990.99, and a pixel stops once its accumulated opacity would exceed 0.99990.9999.
  5. 5The backward pass traverses the same lists back to front. Intermediate transmittances are recovered from the final one, so there is no cap on how many splats can receive gradient from a pixel.

Optimisation and adaptive density control

Training renders a training view, compares it with the photograph, and steps all parameters with Adam. The loss is L=(1−λ)L1+λLD-SSIM\mathcal L=(1-\lambda)\mathcal L_1+\lambda\mathcal L_{\text{D-SSIM}} with λ=0.2\lambda=0.2. The SfM points give a good but sparse start. Initial covariances are isotropic, with radius equal to the mean distance to the three nearest points. Gradient descent alone cannot add primitives where geometry is missing, so every 100 iterations the method edits the set:

Adaptive density control (Kerbl et al. 2023, Sec. 5.2)

Trigger
average magnitude of the view-space positional gradient above τpos=0.0002\tau_{\text{pos}}=0.0002: the Gaussian is being pulled hard, so the region is badly reconstructed
Clone
under-reconstruction (a small Gaussian): copy it and move the copy along the positional gradient, so volume and count both grow
Split
over-reconstruction (a large Gaussian covering detail): replace it by two Gaussians with scales divided by ϕ=1.6\phi=1.6 and means sampled from the original Gaussian as a PDF
Prune
remove Gaussians with opacity below ϵα\epsilon_\alpha (0.005 in the code) and ones that are very large in world or view space
Opacity reset
every 3000 iterations set all oo close to zero; the optimiser raises the useful ones again and pruning removes the rest, which controls floaters near the cameras
Schedule
30K iterations; SH degree raised by one band every 1000 iterations starting from degree 0; a short warm-up at 4× lower resolution
Σ = RSSᵀRᵀ → Σ′ = JWΣWᵀJᵀ → sort by depth → C = Σ cᵢαᵢ Π(1 − αⱼ) · click a pixel57766876565432×24 px · 8×8 tiles, corner = list length · - - 3σ extent of G3top view (x, z)● camera at r = 4yaw 30°, looking at 0Gaussians 9in view 9sort keys K 72α-blends 1697per pixel 2.2K = Σ tiles touchedkey = (tile id, depth)■ G3: s = (0.55, 0.07, 0.07), o = 0.85view depth z 4.062-D mean (px) (13.0, 7.1)Σ′ (px²) [15.2 11.1; 11.1 9.4]3σ radius, tiles 15 px, 12pixel (14, 9), tile (1, 1): front to backsplatzαᵢTᵢTᵢαᵢG23.590.1271.0000.127G34.060.3810.8730.332G04.210.1010.5410.055background T_final 0.486 · Σ Tᵢαᵢ + T 1.000pixel colour. Same weights asNeRF: αᵢ ↔ 1 − exp(−σᵢδᵢ)α = min(0.99, o·exp(−½dᵀΣ′⁻¹d)), skip α < 1/255,stop before T drops below 10⁻⁴; Σ′ += 0.3 px²·Isplit: two copies, s / 1.6, means ~ N(μ, Σ)prune: drop Gaussians with o < 0.005
Gaussian
density ctrl
fig.Nine anisotropic 3-D Gaussians, each with one colour (degree-0 SH), rendered at 32 × 24 px with 8 × 8 tiles (16 × 16 in the paper) by a camera 4 units from the origin. At the default yaw of 30° all 9 are in view. The tile lists (corner numbers) hold 4 to 8 splats, the frame needs K = 72 (tile, depth) sort keys, and it performs 1697 α-blends, 2.2 per pixel. G3 is a needle with scales (0.55, 0.07, 0.07) rotated by 40°. It sits at depth 4.06 with 2-D mean (13.0, 7.1) and Σ′ = [15.2 11.1; 11.1 9.4] px². Its 3σ radius of 15 px puts it in all 12 tiles. At the marked pixel (14, 9) three splats contribute front to back. G2 has α = 0.127 at T = 1. G3 has α = 0.381 at T = 0.873 (weight 0.332). G0 has α = 0.101 at T = 0.541. The background keeps T = 0.486. Rotate G3 to 90°: Σ′ becomes [0.8 0.7; 0.7 28.6], a vertical needle. Its α at the pixel falls to 0.202 and T rises to 0.626. Press split: G3 becomes two copies with scales / 1.6 = (0.34, 0.04, 0.04) and means drawn from G3 itself. K rises to 75, and the copy G3a has a 10-px radius and touches 9 tiles. Reset, set G3's opacity to 0 and prune: 8 Gaussians, K = 60, 1568 blends. Turn the yaw and the depth order changes. At 150°, G3 is the front splat at the pixel with α = 0.823.

Worked example: one Gaussian, one pixel

A Gaussian with scales s=(0.5,0.1,0.1)s=(0.5,0.1,0.1) rotated 30° about the zz axis, at camera-space position (0,0,4)(0,0,4). Take W=IW=I, focal length f=100f=100 px and opacity o=0.8o=0.8. The 0.30.3 px² filter is omitted. On the optical axis x=y=0x=y=0, so J=(25000250)J=\begin{pmatrix}25&0&0\\0&25&0\end{pmatrix}.

quantityvalue
Σ = R diag(0.25, 0.01, 0.01) Rᵀ[[0.19, 0.1039, 0], [0.1039, 0.07, 0], [0, 0, 0.01]]
Σ′ = JΣJᵀ = 625 · (upper-left 2×2)[[118.75, 64.95], [64.95, 43.75]] px²
eigenvalues of Σ′ (std devs)156.25 and 6.25 px² (12.5 px and 2.5 px)
3σ radius⌈3 · 12.5⌉ = 38 px
conic Σ′⁻¹[[0.0448, −0.0665], [−0.0665, 0.1216]]
α at offset (0, 0)0.8
α at offset (10, 0)0.8 · e^(−2.24) = 0.0852
α at offset (0, 5)0.8 · e^(−1.52) = 0.1750
α at offset (10, 6), near the major axis0.8 · e^(−0.4382) = 0.5162
Projection and per-pixel opacity, computed in Python

Compositing two splats at offset (10, 6)

  1. 1The projection scales Σ\Sigma by (f/z)2=625(f/z)^2=625, so the 3-D standard deviations 0.50.5 and 0.10.1 become 12.512.5 px and 2.52.5 px. The rotation is kept: the splat's major axis points along (cos⁡30°,sin⁡30°)(\cos30°,\sin30°).
  2. 2Pixel offset (10,0)(10,0) is 10 px from the centre but off the major axis, so α\alpha is only 0.0850.085. (10,6)(10,6) is farther away (11.7 px) but nearly on the axis, so α=0.516\alpha=0.516. Anisotropy is what lets one Gaussian cover an edge or a thin structure.
  3. 3Behind it is a second splat with αB=0.6\alpha_B=0.6 at this pixel. Front: wA=αA=0.5162w_A=\alpha_A=0.5162. Back: wB=(1−0.5162)⋅0.6=0.2903w_B=(1-0.5162)\cdot0.6=0.2903. Remaining transmittance T=0.4838⋅0.4=0.1935T=0.4838\cdot0.4=0.1935, and wA+wB+T=1w_A+w_B+T=1.
  4. 4With colours cA=(1,0.2,0.2)\mathbf c_A=(1,0.2,0.2) and cB=(0.2,0.4,1)\mathbf c_B=(0.2,0.4,1): C=0.5162 cA+0.2903 cB=(0.5743, 0.2194, 0.3935)C=0.5162\,\mathbf c_A+0.2903\,\mathbf c_B=(0.5743,\,0.2194,\,0.3935), plus 0.19350.1935 times the background.
python
1import torch, torch.nn.functional as F
2
3def quat_to_rot(q):
4 # q (N, 4) = (w, x, y, z), normalised here so any 4-vector is a valid rotation
5 w, x, y, z = F.normalize(q, dim=-1).unbind(-1)
6 return torch.stack([
7 1 - 2 * (y * y + z * z), 2 * (x * y - w * z), 2 * (x * z + w * y),
8 2 * (x * y + w * z), 1 - 2 * (x * x + z * z), 2 * (y * z - w * x),
9 2 * (x * z - w * y), 2 * (y * z + w * x), 1 - 2 * (x * x + y * y)], -1).view(-1, 3, 3)
10
11def render(mu, log_s, q, logit_o, rgb, Wc, tc, f, H, W):
12 # mu (N,3) world means, Wc (3,3) world->camera rotation, tc (3,) translation, f focal in pixels
13 R = quat_to_rot(q)
14 M = R * log_s.exp()[:, None, :] # R S
15 Sigma = M @ M.transpose(1, 2) # (N,3,3) Sigma = R S S^T R^T
16 p = mu @ Wc.T + tc # camera-space means (N,3)
17 x, y, z = p.unbind(-1)
18 J = torch.zeros(len(mu), 2, 3)
19 J[:, 0, 0] = f / z; J[:, 0, 2] = -f * x / z**2 # Jacobian of the perspective projection
20 J[:, 1, 1] = f / z; J[:, 1, 2] = -f * y / z**2
21 T = J @ Wc # (N,2,3)
22 S2 = T @ Sigma @ T.transpose(1, 2) + 0.3 * torch.eye(2) # Sigma' = J W Sigma W^T J^T (+ low-pass)
23 uv = torch.stack([f * x / z + W / 2, f * y / z + H / 2], -1)
24 order = z.argsort() # front to back (one global sort; 3DGS sorts per tile)
25 ys, xs = torch.meshgrid(torch.arange(H) + 0.5, torch.arange(W) + 0.5, indexing="ij")
26 pix = torch.stack([xs, ys], -1).view(-1, 1, 2) # (P,1,2)
27 d = pix - uv[order][None] # (P,N,2)
28 power = -0.5 * (d[..., None, :] @ torch.linalg.inv(S2[order]) @ d[..., None]).squeeze(-1).squeeze(-1)
29 alpha = (torch.sigmoid(logit_o[order]) * power.exp()).clamp(max=0.99) # (P,N)
30 T_i = torch.cumprod(torch.cat([torch.ones_like(alpha[:, :1]), 1 - alpha], 1), 1) # (P,N+1)
31 w = alpha * T_i[:, :-1] # same weights as NeRF: T_i * alpha_i
32 img = w @ torch.sigmoid(rgb[order]) # degree-0 colour only (no SH)
33 return img.view(H, W, 3), T_i[:, -1].view(H, W), uv
34
35def densify(params, grad2d, tau=2e-4, scene_extent=1.0, pct=0.01):
36 # adaptive density control: large view-space gradient -> clone (small Gaussians) or split (large ones).
37 # 3DGS averages the 2-D gradient over views every 100 iterations and uses tau = 0.0002 in NDC units.
38 big_grad = grad2d > tau
39 large = params["log_s"].exp().max(-1).values > pct * scene_extent
40 clone, split = big_grad & ~large, big_grad & large
41 R = quat_to_rot(params["q"][split])
42 s = params["log_s"][split].exp()
43 new_mu = params["mu"][split][None] + (R @ (torch.randn(2, *s.shape) * s)[..., None]).squeeze(-1) # 2 draws ~ N(mu, Sigma)
44 out = {}
45 for k, v in params.items():
46 keep = v[~split].detach()
47 add_split = v[split].detach().repeat(2, *[1] * (v.dim() - 1))
48 if k == "mu": add_split = new_mu.reshape(-1, 3).detach()
49 if k == "log_s": add_split = add_split - torch.log(torch.tensor(1.6)) # scale / 1.6
50 out[k] = torch.cat([keep, v[clone].detach(), add_split]).requires_grad_(True)
51 return out, int(clone.sum()), int(split.sum())
52
53torch.manual_seed(0)
54N, H, W = 200, 48, 64
55params = dict(mu=torch.randn(N, 3) * 0.6, log_s=torch.full((N, 3), -2.5), q=torch.randn(N, 4),
56 logit_o=torch.zeros(N), rgb=torch.randn(N, 3))
57for v in params.values(): v.requires_grad_(True)
58Wc, tc = torch.eye(3), torch.tensor([0.0, 0.0, 4.0]) # camera 4 units in front of the origin
59target = torch.rand(H, W, 3)
60img, T, uv = render(**params, Wc=Wc, tc=tc, f=40.0, H=H, W=W)
61uv.retain_grad()
62loss = (img - target).abs().mean() # 3DGS uses 0.8 * L1 + 0.2 * D-SSIM
63loss.backward()
64print(img.shape, T.shape, uv.grad.shape) # torch.Size([48, 64, 3]) torch.Size([48, 64]) torch.Size([200, 2])
65params, n_clone, n_split = densify(params, uv.grad.norm(dim=-1), tau=1e-4)
66print(n_clone, n_split, params["mu"].shape[0]) # 0 20 220 with this seed: all are "large", so 20 splits, 200 - 20 + 2*20
Differentiable Gaussian splatting in plain PyTorch (dense, no tiles), plus clone/split densification. Real implementations use a CUDA tile rasteriser.

Results

methodPSNR ↑SSIM ↑LPIPS ↓trainFPSmemory
Instant-NGP (big)25.590.6990.3317.5 min9.4348 MB
Mip-NeRF 36027.690.7920.23748 h0.068.6 MB
3DGS-7K25.600.7700.2796.4 min160523 MB
3DGS-30K27.210.8150.21441.6 min134734 MB
Mip-NeRF 360 dataset, Kerbl et al. (2023) Table 1. FPS on an NVIDIA A6000; 3DGS-7K and -30K are the same method stopped at 7K and 30K iterations.

The abstract's target is real-time rendering, at least 30 fps at 1080p. The measured frame rates are well above it, and three to four orders of magnitude faster than Mip-NeRF 360 at similar quality: Mip-NeRF 360 keeps a small lead in PSNR, while 3DGS is better in SSIM and LPIPS. Memory is the price, hundreds of megabytes of Gaussians against a few megabytes of MLP. Much follow-up work (quantisation, pruning, SH distillation) compresses 3DGS scenes by an order of magnitude.

Use when

  • +Real-time rendering: 134 fps on Mip-NeRF 360 scenes versus 0.06 fps for Mip-NeRF 360, at comparable quality
  • +Training in tens of minutes instead of hours or days
  • +Explicit primitives: no computation in empty space, easy editing, compositing and export
  • +The same compositing equation as NeRF, so the photometric training recipe carries over

Avoid when

  • −Large memory: hundreds of MB per scene (59 floats per Gaussian, millions of Gaussians)
  • −Approximate depth ordering causes popping; aliasing when zooming far from the training scales
  • −Relies on SfM initialisation and densification heuristics; sparse views give floaters
  • −Not a smooth continuous field: surfaces and normals are hard to extract without extra regularisation

Check yourself

Why does 3DGS parametrise each covariance as Σ = RSSᵀRᵀ instead of optimising the six entries of Σ directly?

Check yourself

A pixel is covered by three splats front to back with α = 0.5, 0.5, 0.5. What weight does the third splat get?

Remember

  • ✓3DGS scene: anisotropic Gaussians with mean μ\boldsymbol\mu, covariance Σ=RSS⊤R⊤\Sigma=RSS^\top R^\top, opacity oo and degree-3 SH colour (59 floats each); projected as Σ′=JWΣW⊤J⊤\Sigma'=JW\Sigma W^\top J^\top.
  • ✓Rendering is NeRF's sum with explicit opacities: C=∑iciαi∏j<i(1−αj)C=\sum_i\mathbf c_i\alpha_i\prod_{j<i}(1-\alpha_j), αi=oiexp⁡(−12d⊤Σi′−1d)\alpha_i=o_i\exp(-\tfrac12\mathbf d^\top\Sigma_i'^{-1}\mathbf d), splats sorted by depth per 16×16 tile.
  • ✓Adaptive density control every 100 iterations: clone small and split large (s/1.6s/1.6) Gaussians with view-space gradient above 0.0002, prune transparent ones, reset opacity every 3000 iterations.
  • ✓Initialised from SfM points; trained with 0.8·L1 + 0.2·D-SSIM for 30K iterations.
  • ✓Mip-NeRF 360 scenes: 27.21 PSNR at 134 fps after about 41 min of training, against 27.69 PSNR at 0.06 fps after 48 h for Mip-NeRF 360.
← back to Deep Learning roadmap