Definition
Neural radiance field (NeRF) A neural radiance field is a continuous scene representation . An MLP maps a 3-D position and a viewing direction to an emitted colour and a volume density . It was introduced by Mildenhall, Srinivasan, Tancik, Barron, Ramamoorthi and Ng, "NeRF: Representing Scenes as Neural Radiance Fields for View Synthesis" (ECCV 2020). Images are produced by classical volume rendering along camera rays. One network is optimised per scene, from a few dozen to a few hundred photographs with known camera poses. It then synthesises the scene from viewpoints that were never photographed (novel-view synthesis).
Novel-view synthesis is a generative problem: produce plausible images of a scene from new cameras. Before NeRF the strongest methods stored scenes in discrete form: voxel grids (Neural Volumes, 2019), multiplane images (Local Light Field Fusion, 2019), or an MLP that outputs features for a learned ray-marcher (Scene Representation Networks, 2019). Grids cost memory cubic in resolution. NeRF stores the scene in the weights of a plain MLP of about 5 MB and renders by integrating the field along rays. Three choices made it work: a differentiable volume-rendering quadrature, a positional encoding of the inputs, and hierarchical sampling of points along each ray. On the paper's synthetic 360° scenes it beat the previous best by more than 4 dB PSNR. NeRF is not a sample-from-noise generator like a Generative adversarial network (GAN) or Diffusion models. Generative 3-D methods, though, build on it, as the last section shows.
The field and the rendering integral
A pixel defines a ray from the camera centre , between near and far bounds and . Density is the differential probability that the ray terminates at . The expected colour is
- volume density; depends on position only, so geometry is the same from every view
- emitted RGB; depends on position and direction, so specular highlights can move with the camera
- accumulated transmittance: the probability that the ray reaches without hitting anything
- density of the ray's termination depth; it integrates to the pixel's opacity
The integral is estimated from samples with spacings , using the quadrature rule from Max's 1995 review of volume rendering. Within each interval the density and colour are treated as constant:
- opacity of interval : the chance the ray stops inside it, given it got there
- compositing weight of sample ; and the rest is background
- distance to the next sample; the reference code sets the last one to so the far end is opaque
- expected termination depth, a free depth map
This is front-to-back alpha compositing with derived from density. Every operation is differentiable in and , so the gradient of a pixel error flows back into the MLP at every sample. The training loss is the squared error between rendered and photographed pixels over a batch of rays:
- ground-truth colour of the pixel the ray passes through
- renderings by the coarse and fine networks (next sections); both are supervised
Positional encoding and spectral bias
An MLP fed raw renders blurry images. Neural networks are biased toward low-frequency functions (Rahaman et al. 2019, "spectral bias"), and a scene has sharp edges and fine texture. NeRF passes each coordinate through a fixed map into sines and cosines of geometrically increasing frequency:
- one coordinate, normalised to ; is applied to each of the 3 components separately
- number of octaves: , (the reference code also appends the raw input, giving 63 and 27)
- highest frequency; it sets the finest detail the field can easily express
This is the same idea as the sinusoidal position code of the Transformer, applied to continuous coordinates. Tancik et al. (NeurIPS 2020, "Fourier Features Let Networks Learn High Frequency Functions") explained why it works. Through the neural tangent kernel, a Fourier feature map turns the MLP's kernel into a stationary one whose bandwidth is set by the frequencies. Too few octaves gives blur. Too many gives noisy, aliased fits between training views.
Network and view dependence
The MLP has 8 fully connected ReLU layers of 256 channels. is concatenated back in at the fifth layer (a skip connection). The trunk outputs (through a ReLU) and a 256-d feature. The feature is concatenated with and passed through one 128-channel layer to a sigmoid RGB. Because is computed before the direction enters, the geometry is forced to be multi-view consistent. Only colour can vary with view, which is how NeRF reproduces specular reflections. In the paper's ablation on the synthetic scenes, removing positional encoding cost 2.2 dB PSNR (31.01 → 28.77) and removing view dependence 3.4 dB (→ 27.66).
Hierarchical sampling: 64 coarse + 128 fine
Evaluating the MLP densely along every ray wastes most queries on empty space and occluded regions. NeRF trains two networks. The coarse network is evaluated at stratified points. Split into equal bins and draw one uniform sample in each, so that over training the network sees continuous positions, not a fixed grid. Its weights, normalised to , define a piecewise-constant PDF along the ray. more points are drawn from it by inverse-transform sampling. The fine network is then evaluated at all points, and is the final rendering. The fine samples land where the coarse pass found surfaces. This is importance sampling of the rendering integral.
Setup in the paper
- Data
- Diffuse Synthetic 360°, Realistic Synthetic 360° (8 Blender scenes, 100 training views at 800×800), and Real Forward-Facing (8 LLFF scenes, 20–62 phone images, poses from COLMAP)
- Network
- 8×256 ReLU MLP, skip at layer 5, one 128-channel view-dependent layer; for , for
- Sampling
- coarse + fine samples per ray; coarse and fine networks both trained
- Optimisation
- batch of 4096 rays, Adam with learning rate decaying exponentially to
- Cost
- 100k–300k iterations, about 1–2 days per scene on one NVIDIA V100
- Storage
- about 5 MB of network weights per scene, versus over 15 GB of sampled volumes for LLFF on the synthetic scenes
Worked example: four samples on one ray
One grey channel. Samples at with for all four (no infinite last interval here), densities and colours .
| i | tᵢ | σᵢδᵢ | Tᵢ | αᵢ = 1 − e^(−σᵢδᵢ) | wᵢ = Tᵢαᵢ | wᵢcᵢ | ŵᵢ (fine PDF) |
|---|---|---|---|---|---|---|---|
| 1 | 1.0 | 0.0 | 1.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 |
| 2 | 1.5 | 0.5 | 1.0000 | 0.3935 | 0.3935 | 0.3541 | 0.4057 |
| 3 | 2.0 | 2.0 | 0.6065 | 0.8647 | 0.5244 | 0.2622 | 0.5408 |
| 4 | 2.5 | 1.0 | 0.0821 | 0.6321 | 0.0519 | 0.0052 | 0.0535 |
Reading the table
- 1Transmittance accumulates optical depth: , . Sample 4 is mostly hidden behind sample 3.
- 2Pixel value . Opacity , so the background shows through with weight .
- 3Expected depth , or after dividing by the opacity. The highest weight is at , where peaks.
- 4For hierarchical sampling, the normalised weights are the PDF. About 95% of the fine samples go to the bins around and , and none to .
- 5Positional encoding of one coordinate with : pairs , , for frequencies . A shift of 0.1 in rotates the third pair by , so the last octaves are what resolve fine detail.
1import torch, torch.nn as nn, torch.nn.functional as F23def posenc(p, L):4 # p (..., 3) -> (..., 3 + 6L): [p, sin(2^k pi p), cos(2^k pi p)] for k < L5 freqs = (2.0 ** torch.arange(L)) * torch.pi # (L,)6 ang = p[..., None, :] * freqs[:, None] # (..., L, 3)7 return torch.cat([p, ang.sin().flatten(-2), ang.cos().flatten(-2)], -1)89class NeRF(nn.Module):10 def __init__(self, Lx=10, Ld=4, W=256):11 super().__init__()12 self.Lx, self.Ld = Lx, Ld13 dx, dd = 3 + 6 * Lx, 3 + 6 * Ld # 63, 2714 self.l1 = nn.ModuleList([nn.Linear(dx, W)] + [nn.Linear(W, W) for _ in range(4)])15 self.l2 = nn.ModuleList([nn.Linear(W + dx, W)] + [nn.Linear(W, W) for _ in range(2)]) # skip into layer 516 self.sigma = nn.Linear(W, 1)17 self.feat = nn.Linear(W, W)18 self.rgb = nn.Sequential(nn.Linear(W + dd, W // 2), nn.ReLU(), nn.Linear(W // 2, 3), nn.Sigmoid())1920 def forward(self, x, d):21 gx, gd = posenc(x, self.Lx), posenc(d, self.Ld)22 h = gx23 for l in self.l1: h = F.relu(l(h))24 h = torch.cat([h, gx], -1)25 for l in self.l2: h = F.relu(l(h))26 sigma = F.relu(self.sigma(h)).squeeze(-1) # density depends on x only27 rgb = self.rgb(torch.cat([self.feat(h), gd], -1)) # colour also sees the view direction28 return rgb, sigma2930def composite(rgb, sigma, t):31 # rgb (R, N, 3), sigma (R, N), t (R, N) sorted sample depths32 delta = torch.cat([t[:, 1:] - t[:, :-1], torch.full_like(t[:, :1], 1e10)], -1)33 alpha = 1 - torch.exp(-sigma * delta)34 T = torch.cumprod(torch.cat([torch.ones_like(alpha[:, :1]), 1 - alpha + 1e-10], -1), -1)[:, :-1]35 w = T * alpha # (R, N)36 return (w[..., None] * rgb).sum(1), w3738def sample_pdf(t, w, n):39 # inverse-transform sampling from the piecewise-constant PDF of the coarse weights40 mids = 0.5 * (t[:, 1:] + t[:, :-1]) # (R, N-1) bin edges41 pdf = (w[:, 1:-1] + 1e-5); pdf = pdf / pdf.sum(-1, keepdim=True)42 cdf = torch.cat([torch.zeros_like(pdf[:, :1]), pdf.cumsum(-1)], -1) # (R, N-1)43 u = torch.rand(t.shape[0], n)44 i = torch.searchsorted(cdf, u, right=True).clamp(1, cdf.shape[-1] - 1)45 c0, c1 = cdf.gather(-1, i - 1), cdf.gather(-1, i)46 b0, b1 = mids.gather(-1, i - 1), mids.gather(-1, i)47 return b0 + (u - c0) / (c1 - c0).clamp_min(1e-5) * (b1 - b0)4849def render(coarse, fine, o, d, near=2.0, far=6.0, Nc=64, Nf=128):50 R = o.shape[0]51 t = near + (far - near) * (torch.arange(Nc) + torch.rand(R, Nc)) / Nc # stratified52 x = o[:, None] + t[..., None] * d[:, None]53 rgb_c, w = composite(*coarse(x, d[:, None].expand_as(x)), t)54 t_f = torch.sort(torch.cat([t, sample_pdf(t, w.detach(), Nf)], -1), -1).values # Nc + Nf55 x = o[:, None] + t_f[..., None] * d[:, None]56 rgb_f, _ = composite(*fine(x, d[:, None].expand_as(x)), t_f)57 return rgb_c, rgb_f5859coarse, fine = NeRF(), NeRF()60print(sum(p.numel() for p in coarse.parameters())) # 595844 parameters per network61o = torch.zeros(1024, 3); d = F.normalize(torch.randn(1024, 3), dim=-1)62target = torch.rand(1024, 3)63rgb_c, rgb_f = render(coarse, fine, o, d)64loss = ((rgb_c - target) ** 2).sum(-1).mean() + ((rgb_f - target) ** 2).sum(-1).mean()65loss.backward()66print(rgb_c.shape, rgb_f.shape, posenc(o, 10).shape, posenc(d, 4).shape)67# torch.Size([1024, 3]) torch.Size([1024, 3]) torch.Size([1024, 63]) torch.Size([1024, 27])Results
| method | Realistic Synthetic 360° | Real Forward-Facing (LLFF) |
|---|---|---|
| SRN | 22.26 / 0.846 / 0.170 | 22.84 / 0.668 / 0.378 |
| Neural Volumes | 26.05 / 0.893 / 0.160 | — |
| LLFF | 24.88 / 0.911 / 0.114 | 24.13 / 0.798 / 0.212 |
| NeRF | 31.01 / 0.947 / 0.081 | 26.50 / 0.811 / 0.250 |
Use when
- +Continuous, compact scene: about 5 MB of weights instead of gigabytes of voxels or MPIs
- +Photorealistic novel views with view-dependent effects; geometry and depth come for free from σ
- +Fully differentiable, so the renderer can sit inside larger learning systems (pose refinement, text-to-3D, 3-D-aware GANs)
- +No 3-D supervision: only posed 2-D images
Avoid when
- −Slow: 192 MLP evaluations per pixel, far from real-time rendering, and 1–2 days of training per scene in the original
- −One network per scene, no generalisation across scenes without extra machinery
- −Needs accurate poses, a static scene and fixed lighting; prone to floaters with sparse views
- −Aliasing when rendering at resolutions or distances unlike the training views (addressed by mip-NeRF)
Check yourself
Why does the NeRF MLP output σ before the viewing direction d is fed in?
Check yourself
Two samples on a ray have σδ = 0.5 and then σδ = 2.0. What is the compositing weight of the second sample?
Remember
- ✓NeRF rendering: with ; the MLP is fitted per scene with .
- ✓Positional encoding with for position and for direction overcomes the MLP's spectral bias toward smooth functions.
- ✓Hierarchical sampling: 64 stratified coarse samples; their normalised weights form a PDF for 128 fine samples; the fine network renders all 192.
- ✓ depends on only and on , which gives consistent geometry and view-dependent appearance.
- ✓Realistic Synthetic 360°: 31.01 dB PSNR versus 26.05 for the best prior method; cost about 1–2 days of training per scene on a V100.
- ✓Successors: Instant-NGP hash grids (minutes), mip-NeRF cones (anti-aliasing), DreamFusion's score distillation (text-to-3D), 3D Gaussian splatting (real time).