Neural radiance fields (NeRF)

A NeRF stores one scene in an MLP F(x, d) → (colour, density). A pixel is rendered by alpha-compositing samples along its camera ray, C = Σ Tᵢ(1 − exp(−σᵢδᵢ))cᵢ. Because this is differentiable, the MLP can be fitted to posed photographs with a squared colour error, and then renders novel views.

15 min read1378 wordsadvanced0 viewsupdated 2026-10-12

Definition

Neural radiance field (NeRF) A neural radiance field is a continuous scene representation FΘ:(x,d)↦(c,σ)F_\Theta:(\mathbf x,\mathbf d)\mapsto(\mathbf c,\sigma). An MLP maps a 3-D position x\mathbf x and a viewing direction d\mathbf d to an emitted colour c\mathbf c and a volume density σ≥0\sigma\ge0. It was introduced by Mildenhall, Srinivasan, Tancik, Barron, Ramamoorthi and Ng, "NeRF: Representing Scenes as Neural Radiance Fields for View Synthesis" (ECCV 2020). Images are produced by classical volume rendering along camera rays. One network is optimised per scene, from a few dozen to a few hundred photographs with known camera poses. It then synthesises the scene from viewpoints that were never photographed (novel-view synthesis).

Novel-view synthesis is a generative problem: produce plausible images of a scene from new cameras. Before NeRF the strongest methods stored scenes in discrete form: voxel grids (Neural Volumes, 2019), multiplane images (Local Light Field Fusion, 2019), or an MLP that outputs features for a learned ray-marcher (Scene Representation Networks, 2019). Grids cost memory cubic in resolution. NeRF stores the scene in the weights of a plain MLP of about 5 MB and renders by integrating the field along rays. Three choices made it work: a differentiable volume-rendering quadrature, a positional encoding of the inputs, and hierarchical sampling of points along each ray. On the paper's synthetic 360° scenes it beat the previous best by more than 4 dB PSNR. NeRF is not a sample-from-noise generator like a Generative adversarial network (GAN) or Diffusion models. Generative 3-D methods, though, build on it, as the last section shows.

The field and the rendering integral

A pixel defines a ray r(t)=o+td\mathbf r(t)=\mathbf o+t\mathbf d from the camera centre o\mathbf o, between near and far bounds tnt_n and tft_f. Density σ(x)\sigma(\mathbf x) is the differential probability that the ray terminates at x\mathbf x. The expected colour is

C(r)=∫tntfT(t) σ(r(t)) c(r(t),d) dt,T(t)=exp⁡ ⁣(−∫tntσ(r(s)) ds)C(\mathbf r)=\int_{t_n}^{t_f}T(t)\,\sigma(\mathbf r(t))\,\mathbf c(\mathbf r(t),\mathbf d)\,dt,\qquad T(t)=\exp\!\Big(-\int_{t_n}^{t}\sigma(\mathbf r(s))\,ds\Big)
σ(x)\sigma(\mathbf x)
volume density; depends on position only, so geometry is the same from every view
c(x,d)\mathbf c(\mathbf x,\mathbf d)
emitted RGB; depends on position and direction, so specular highlights can move with the camera
T(t)T(t)
accumulated transmittance: the probability that the ray reaches tt without hitting anything
T(t)σ(t)T(t)\sigma(t)
density of the ray's termination depth; it integrates to the pixel's opacity

The integral is estimated from NN samples t1<⋯<tNt_1<\dots<t_N with spacings δi=ti+1−ti\delta_i=t_{i+1}-t_i, using the quadrature rule from Max's 1995 review of volume rendering. Within each interval the density and colour are treated as constant:

C^(r)=∑i=1NTi (1−e−σiδi) ci,Ti=exp⁡ ⁣(−∑j<iσjδj)=∏j<i(1−αj),αi=1−e−σiδi\hat C(\mathbf r)=\sum_{i=1}^{N}T_i\,\big(1-e^{-\sigma_i\delta_i}\big)\,\mathbf c_i,\qquad T_i=\exp\!\Big(-\sum_{j<i}\sigma_j\delta_j\Big)=\prod_{j<i}(1-\alpha_j),\qquad \alpha_i=1-e^{-\sigma_i\delta_i}
αi\alpha_i
opacity of interval ii: the chance the ray stops inside it, given it got there
wi=Tiαiw_i=T_i\alpha_i
compositing weight of sample ii; ∑iwi=1−TN+1≤1\sum_iw_i=1-T_{N+1}\le1 and the rest is background
δi\delta_i
distance to the next sample; the reference code sets the last one to 101010^{10} so the far end is opaque
∑iwiti\sum_iw_it_i
expected termination depth, a free depth map

This is front-to-back alpha compositing with αi\alpha_i derived from density. Every operation is differentiable in σi\sigma_i and ci\mathbf c_i, so the gradient of a pixel error flows back into the MLP at every sample. The training loss is the squared error between rendered and photographed pixels over a batch R\mathcal R of rays:

L=∑r∈R[∥C^c(r)−C(r)∥22+∥C^f(r)−C(r)∥22]\mathcal L=\sum_{\mathbf r\in\mathcal R}\Big[\big\|\hat C_c(\mathbf r)-C(\mathbf r)\big\|_2^2+\big\|\hat C_f(\mathbf r)-C(\mathbf r)\big\|_2^2\Big]
C(r)C(\mathbf r)
ground-truth colour of the pixel the ray passes through
C^c, C^f\hat C_c,\ \hat C_f
renderings by the coarse and fine networks (next sections); both are supervised

Positional encoding and spectral bias

An MLP fed raw (x,y,z,θ,ϕ)(x,y,z,\theta,\phi) renders blurry images. Neural networks are biased toward low-frequency functions (Rahaman et al. 2019, "spectral bias"), and a scene has sharp edges and fine texture. NeRF passes each coordinate through a fixed map into sines and cosines of geometrically increasing frequency:

γ(p)=(sin⁡(20πp), cos⁡(20πp), …, sin⁡(2L−1πp), cos⁡(2L−1πp)),L=10 for x,L=4 for d\gamma(p)=\big(\sin(2^0\pi p),\ \cos(2^0\pi p),\ \dots,\ \sin(2^{L-1}\pi p),\ \cos(2^{L-1}\pi p)\big),\qquad L=10\ \text{for }\mathbf x,\quad L=4\ \text{for }\mathbf d
pp
one coordinate, normalised to [−1,1][-1,1]; γ\gamma is applied to each of the 3 components separately
LL
number of octaves: γ(x)∈R60\gamma(\mathbf x)\in\mathbb R^{60}, γ(d)∈R24\gamma(\mathbf d)\in\mathbb R^{24} (the reference code also appends the raw input, giving 63 and 27)
2L−1π2^{L-1}\pi
highest frequency; it sets the finest detail the field can easily express

This is the same idea as the sinusoidal position code of the Transformer, applied to continuous coordinates. Tancik et al. (NeurIPS 2020, "Fourier Features Let Networks Learn High Frequency Functions") explained why it works. Through the neural tangent kernel, a Fourier feature map turns the MLP's kernel into a stationary one whose bandwidth is set by the frequencies. Too few octaves gives blur. Too many gives noisy, aliased fits between training views.

Network and view dependence

The MLP has 8 fully connected ReLU layers of 256 channels. γ(x)\gamma(\mathbf x) is concatenated back in at the fifth layer (a skip connection). The trunk outputs σ\sigma (through a ReLU) and a 256-d feature. The feature is concatenated with γ(d)\gamma(\mathbf d) and passed through one 128-channel layer to a sigmoid RGB. Because σ\sigma is computed before the direction enters, the geometry is forced to be multi-view consistent. Only colour can vary with view, which is how NeRF reproduces specular reflections. In the paper's ablation on the synthetic scenes, removing positional encoding cost 2.2 dB PSNR (31.01 → 28.77) and removing view dependence 3.4 dB (→ 27.66).

Hierarchical sampling: 64 coarse + 128 fine

Evaluating the MLP densely along every ray wastes most queries on empty space and occluded regions. NeRF trains two networks. The coarse network is evaluated at Nc=64N_c=64 stratified points. Split [tn,tf][t_n,t_f] into NcN_c equal bins and draw one uniform sample in each, so that over training the network sees continuous positions, not a fixed grid. Its weights, normalised to w^i=wi/∑jwj\hat w_i=w_i/\sum_jw_j, define a piecewise-constant PDF along the ray. Nf=128N_f=128 more points are drawn from it by inverse-transform sampling. The fine network is then evaluated at all Nc+Nf=192N_c+N_f=192 points, and C^f\hat C_f is the final rendering. The fine samples land where the coarse pass found surfaces. This is importance sampling of the rendering integral.

Setup in the paper

Data
Diffuse Synthetic 360°, Realistic Synthetic 360° (8 Blender scenes, 100 training views at 800×800), and Real Forward-Facing (8 LLFF scenes, 20–62 phone images, poses from COLMAP)
Network
8×256 ReLU MLP, skip at layer 5, one 128-channel view-dependent layer; L=10L=10 for x\mathbf x, L=4L=4 for d\mathbf d
Sampling
Nc=64N_c=64 coarse + Nf=128N_f=128 fine samples per ray; coarse and fine networks both trained
Optimisation
batch of 4096 rays, Adam with learning rate 5×10−45\times10^{-4} decaying exponentially to 5×10−55\times10^{-5}
Cost
100k–300k iterations, about 1–2 days per scene on one NVIDIA V100
Storage
about 5 MB of network weights per scene, versus over 15 GB of sampled volumes for LLFF on the synthetic scenes
Ĉ = Σ Tᵢ(1 − e^(−σᵢδᵢ)) cᵢ along r(t) = o + t·d, top-down viewo● coarse ● fine · dot size ∝ √wᵢ■ σ(t) (cap 60) — T(t) estimate · bars wᵢ / max wᵢ100.511.522.5t (near 0.1 → far 2.6)ray angle 18° · seed 1coarse Nc 8 samplescoarse error ‖w − w*‖₁ 1.922fine Nf (inverse CDF) offestimate error 1.922MLP queries 8colour weightestreffog σ=2.50.0000.003solid A σ=300.0590.002solid B σ=300.0000.000thin rod σ=600.0250.984background0.9160.012opacity Σwᵢ 0.084 ref 0.988depth Σwᵢtᵢ 0.113 ref 1.186pixel colourestimate4000-sample refpaper: Nc = 64, Nf = 128,fine net sees all 192 points
coarse Nc
fig.A 2-D scene seen from above: a fog disc (σ = 2.5), two solid discs (σ = 30) and a thin rod 0.07 wide (σ = 60). The camera at o casts one ray, rendered with the NeRF quadrature between near 0.1 and far 2.6. The reference uses 4000 evenly spaced samples. 'error' is the L1 distance between the estimated and reference colour weights of the four objects plus background. At the default 18° the ray hits only the rod. The reference gives it weight 0.984 (opacity 0.988, depth 1.186). Eight stratified coarse samples step over it: opacity 0.084, background weight 0.916, error 1.922. Sixteen coarse samples are still wrong (1.502), and 32 get it right (0.008). Now add Nf = 16 fine samples drawn from the coarse weights. The coarse pass gave the rod only 0.025 of weight, but that is enough to put a PDF peak there. The error drops to 0.012 at 8 + (8 + 16) = 32 MLP queries. Press resample twice (seed 3): the coarse pass sees nothing at the rod, the fine samples go elsewhere, and the error is 1.968. Over 100 jitter seeds at 18°, the run succeeds (error < 0.05) 75 times for 8 + 16, 79 times for 32 uniform samples, and 95 times for 16 + 16. At 0° the ray crosses the fog and clips both solids. There 8 + 16 is worse than 32 uniform samples (mean error 0.172 vs 0.038): the fine samples cluster at the surfaces, while the fog is still integrated with 8 coarse points. With the paper's 64 + 128, every readout is within 0.01 of the reference.

Worked example: four samples on one ray

One grey channel. Samples at t=1.0,1.5,2.0,2.5t=1.0,1.5,2.0,2.5 with δi=0.5\delta_i=0.5 for all four (no infinite last interval here), densities σ=(0,1,4,2)\sigma=(0,1,4,2) and colours c=(0.2,0.9,0.5,0.1)c=(0.2,0.9,0.5,0.1).

itᵢσᵢδᵢTᵢαᵢ = 1 − e^(−σᵢδᵢ)wᵢ = Tᵢαᵢwᵢcᵢŵᵢ (fine PDF)
11.00.01.00000.00000.00000.00000.0000
21.50.51.00000.39350.39350.35410.4057
32.02.00.60650.86470.52440.26220.5408
42.51.00.08210.63210.05190.00520.0535
Volume-rendering quadrature, computed in Python

Reading the table

  1. 1Transmittance accumulates optical depth: T3=e−(0+0.5)=0.6065T_3=e^{-(0+0.5)}=0.6065, T4=e−(0+0.5+2)=0.0821T_4=e^{-(0+0.5+2)}=0.0821. Sample 4 is mostly hidden behind sample 3.
  2. 2Pixel value C^=∑iwici=0.3541+0.2622+0.0052=0.6215\hat C=\sum_iw_ic_i=0.3541+0.2622+0.0052=0.6215. Opacity ∑iwi=0.9698\sum_iw_i=0.9698, so the background shows through with weight T5=0.0302T_5=0.0302.
  3. 3Expected depth ∑iwiti=1.7688\sum_iw_it_i=1.7688, or 1.82391.8239 after dividing by the opacity. The highest weight is at t=2.0t=2.0, where σ\sigma peaks.
  4. 4For hierarchical sampling, the normalised weights w^=(0, 0.406, 0.541, 0.054)\hat w=(0,\,0.406,\,0.541,\,0.054) are the PDF. About 95% of the fine samples go to the bins around t=1.5t=1.5 and t=2.0t=2.0, and none to t=1.0t=1.0.
  5. 5Positional encoding of one coordinate p=0.3p=0.3 with L=3L=3: (sin⁡,cos⁡)(\sin,\cos) pairs (0.809,0.588)(0.809,0.588), (0.951,−0.309)(0.951,-0.309), (−0.588,−0.809)(-0.588,-0.809) for frequencies π,2π,4π\pi,2\pi,4\pi. A shift of 0.1 in pp rotates the third pair by 0.4π0.4\pi, so the last octaves are what resolve fine detail.
python
1import torch, torch.nn as nn, torch.nn.functional as F
2
3def posenc(p, L):
4 # p (..., 3) -> (..., 3 + 6L): [p, sin(2^k pi p), cos(2^k pi p)] for k < L
5 freqs = (2.0 ** torch.arange(L)) * torch.pi # (L,)
6 ang = p[..., None, :] * freqs[:, None] # (..., L, 3)
7 return torch.cat([p, ang.sin().flatten(-2), ang.cos().flatten(-2)], -1)
8
9class NeRF(nn.Module):
10 def __init__(self, Lx=10, Ld=4, W=256):
11 super().__init__()
12 self.Lx, self.Ld = Lx, Ld
13 dx, dd = 3 + 6 * Lx, 3 + 6 * Ld # 63, 27
14 self.l1 = nn.ModuleList([nn.Linear(dx, W)] + [nn.Linear(W, W) for _ in range(4)])
15 self.l2 = nn.ModuleList([nn.Linear(W + dx, W)] + [nn.Linear(W, W) for _ in range(2)]) # skip into layer 5
16 self.sigma = nn.Linear(W, 1)
17 self.feat = nn.Linear(W, W)
18 self.rgb = nn.Sequential(nn.Linear(W + dd, W // 2), nn.ReLU(), nn.Linear(W // 2, 3), nn.Sigmoid())
19
20 def forward(self, x, d):
21 gx, gd = posenc(x, self.Lx), posenc(d, self.Ld)
22 h = gx
23 for l in self.l1: h = F.relu(l(h))
24 h = torch.cat([h, gx], -1)
25 for l in self.l2: h = F.relu(l(h))
26 sigma = F.relu(self.sigma(h)).squeeze(-1) # density depends on x only
27 rgb = self.rgb(torch.cat([self.feat(h), gd], -1)) # colour also sees the view direction
28 return rgb, sigma
29
30def composite(rgb, sigma, t):
31 # rgb (R, N, 3), sigma (R, N), t (R, N) sorted sample depths
32 delta = torch.cat([t[:, 1:] - t[:, :-1], torch.full_like(t[:, :1], 1e10)], -1)
33 alpha = 1 - torch.exp(-sigma * delta)
34 T = torch.cumprod(torch.cat([torch.ones_like(alpha[:, :1]), 1 - alpha + 1e-10], -1), -1)[:, :-1]
35 w = T * alpha # (R, N)
36 return (w[..., None] * rgb).sum(1), w
37
38def sample_pdf(t, w, n):
39 # inverse-transform sampling from the piecewise-constant PDF of the coarse weights
40 mids = 0.5 * (t[:, 1:] + t[:, :-1]) # (R, N-1) bin edges
41 pdf = (w[:, 1:-1] + 1e-5); pdf = pdf / pdf.sum(-1, keepdim=True)
42 cdf = torch.cat([torch.zeros_like(pdf[:, :1]), pdf.cumsum(-1)], -1) # (R, N-1)
43 u = torch.rand(t.shape[0], n)
44 i = torch.searchsorted(cdf, u, right=True).clamp(1, cdf.shape[-1] - 1)
45 c0, c1 = cdf.gather(-1, i - 1), cdf.gather(-1, i)
46 b0, b1 = mids.gather(-1, i - 1), mids.gather(-1, i)
47 return b0 + (u - c0) / (c1 - c0).clamp_min(1e-5) * (b1 - b0)
48
49def render(coarse, fine, o, d, near=2.0, far=6.0, Nc=64, Nf=128):
50 R = o.shape[0]
51 t = near + (far - near) * (torch.arange(Nc) + torch.rand(R, Nc)) / Nc # stratified
52 x = o[:, None] + t[..., None] * d[:, None]
53 rgb_c, w = composite(*coarse(x, d[:, None].expand_as(x)), t)
54 t_f = torch.sort(torch.cat([t, sample_pdf(t, w.detach(), Nf)], -1), -1).values # Nc + Nf
55 x = o[:, None] + t_f[..., None] * d[:, None]
56 rgb_f, _ = composite(*fine(x, d[:, None].expand_as(x)), t_f)
57 return rgb_c, rgb_f
58
59coarse, fine = NeRF(), NeRF()
60print(sum(p.numel() for p in coarse.parameters())) # 595844 parameters per network
61o = torch.zeros(1024, 3); d = F.normalize(torch.randn(1024, 3), dim=-1)
62target = torch.rand(1024, 3)
63rgb_c, rgb_f = render(coarse, fine, o, d)
64loss = ((rgb_c - target) ** 2).sum(-1).mean() + ((rgb_f - target) ** 2).sum(-1).mean()
65loss.backward()
66print(rgb_c.shape, rgb_f.shape, posenc(o, 10).shape, posenc(d, 4).shape)
67# torch.Size([1024, 3]) torch.Size([1024, 3]) torch.Size([1024, 63]) torch.Size([1024, 27])
NeRF MLP, stratified + hierarchical sampling, volume rendering and the two-term loss (runs on CPU)

Results

methodRealistic Synthetic 360°Real Forward-Facing (LLFF)
SRN22.26 / 0.846 / 0.17022.84 / 0.668 / 0.378
Neural Volumes26.05 / 0.893 / 0.160—
LLFF24.88 / 0.911 / 0.11424.13 / 0.798 / 0.212
NeRF31.01 / 0.947 / 0.08126.50 / 0.811 / 0.250
Mildenhall et al. (2020), Table 1: PSNR ↑ / SSIM ↑ / LPIPS ↓

Use when

  • +Continuous, compact scene: about 5 MB of weights instead of gigabytes of voxels or MPIs
  • +Photorealistic novel views with view-dependent effects; geometry and depth come for free from σ
  • +Fully differentiable, so the renderer can sit inside larger learning systems (pose refinement, text-to-3D, 3-D-aware GANs)
  • +No 3-D supervision: only posed 2-D images

Avoid when

  • −Slow: 192 MLP evaluations per pixel, far from real-time rendering, and 1–2 days of training per scene in the original
  • −One network per scene, no generalisation across scenes without extra machinery
  • −Needs accurate poses, a static scene and fixed lighting; prone to floaters with sparse views
  • −Aliasing when rendering at resolutions or distances unlike the training views (addressed by mip-NeRF)

Check yourself

Why does the NeRF MLP output σ before the viewing direction d is fed in?

Check yourself

Two samples on a ray have σδ = 0.5 and then σδ = 2.0. What is the compositing weight of the second sample?

Remember

  • ✓NeRF rendering: C^=∑iTi(1−e−σiδi)ci\hat C=\sum_iT_i(1-e^{-\sigma_i\delta_i})\mathbf c_i with Ti=exp⁡(−∑j<iσjδj)T_i=\exp(-\sum_{j<i}\sigma_j\delta_j); the MLP F(x,d)→(c,σ)F(\mathbf x,\mathbf d)\to(\mathbf c,\sigma) is fitted per scene with ∑r∥C^(r)−C(r)∥2\sum_{\mathbf r}\|\hat C(\mathbf r)-C(\mathbf r)\|^2.
  • ✓Positional encoding γ(p)=(sin⁡2kπp,cos⁡2kπp)k<L\gamma(p)=(\sin 2^k\pi p,\cos 2^k\pi p)_{k<L} with L=10L=10 for position and L=4L=4 for direction overcomes the MLP's spectral bias toward smooth functions.
  • ✓Hierarchical sampling: 64 stratified coarse samples; their normalised weights form a PDF for 128 fine samples; the fine network renders all 192.
  • ✓σ\sigma depends on x\mathbf x only and c\mathbf c on (x,d)(\mathbf x,\mathbf d), which gives consistent geometry and view-dependent appearance.
  • ✓Realistic Synthetic 360°: 31.01 dB PSNR versus 26.05 for the best prior method; cost about 1–2 days of training per scene on a V100.
  • ✓Successors: Instant-NGP hash grids (minutes), mip-NeRF cones (anti-aliasing), DreamFusion's score distillation (text-to-3D), 3D Gaussian splatting (real time).
← back to Deep Learning roadmap