# Neural radiance fields (NeRF)

> A NeRF stores one scene in an MLP F(x, d) → (colour, density). A pixel is rendered by alpha-compositing samples along its camera ray, C = Σ Tᵢ(1 − exp(−σᵢδᵢ))cᵢ. Because this is differentiable, the MLP can be fitted to posed photographs with a squared colour error, and then renders novel views.

Source: https://www.mlfoundry.in/learn/neural-radiance-fields-nerf  
Section: Deep Learning › Generative Deep Learning Algorithms  
Level: advanced · 15 min read · updated 2026-10-12

**Neural radiance field (NeRF):** A **neural radiance field** is a continuous scene representation $F_\Theta:(\mathbf x,\mathbf d)\mapsto(\mathbf c,\sigma)$. An MLP maps a 3-D position $\mathbf x$ and a viewing direction $\mathbf d$ to an emitted colour $\mathbf c$ and a volume density $\sigma\ge0$. It was introduced by Mildenhall, Srinivasan, Tancik, Barron, Ramamoorthi and Ng, "NeRF: Representing Scenes as Neural Radiance Fields for View Synthesis" (ECCV 2020). Images are produced by classical **volume rendering** along camera rays. One network is optimised **per scene**, from a few dozen to a few hundred photographs with known camera poses. It then synthesises the scene from viewpoints that were never photographed (**novel-view synthesis**).

Novel-view synthesis is a generative problem: produce plausible images of a scene from new cameras. Before NeRF the strongest methods stored scenes in discrete form: voxel grids (Neural Volumes, 2019), multiplane images (Local Light Field Fusion, 2019), or an MLP that outputs features for a learned ray-marcher (Scene Representation Networks, 2019). Grids cost memory cubic in resolution. NeRF stores the scene in the weights of a plain MLP of about 5 MB and renders by integrating the field along rays. Three choices made it work: a **differentiable volume-rendering** quadrature, a **positional encoding** of the inputs, and **hierarchical sampling** of points along each ray. On the paper's synthetic 360° scenes it beat the previous best by more than 4 dB PSNR. NeRF is not a sample-from-noise generator like a **Generative adversarial network (GAN)** or **Diffusion models**. Generative 3-D methods, though, build on it, as the last section shows.

## The field and the rendering integral

A pixel defines a ray $\mathbf r(t)=\mathbf o+t\mathbf d$ from the camera centre $\mathbf o$, between near and far bounds $t_n$ and $t_f$. Density $\sigma(\mathbf x)$ is the differential probability that the ray terminates at $\mathbf x$. The expected colour is

$$
C(\mathbf r)=\int_{t_n}^{t_f}T(t)\,\sigma(\mathbf r(t))\,\mathbf c(\mathbf r(t),\mathbf d)\,dt,\qquad T(t)=\exp\!\Big(-\int_{t_n}^{t}\sigma(\mathbf r(s))\,ds\Big)
$$

- $\sigma(\mathbf x)$: volume density; depends on position only, so geometry is the same from every view
- $\mathbf c(\mathbf x,\mathbf d)$: emitted RGB; depends on position **and** direction, so specular highlights can move with the camera
- $T(t)$: accumulated transmittance: the probability that the ray reaches $t$ without hitting anything
- $T(t)\sigma(t)$: density of the ray's termination depth; it integrates to the pixel's opacity

The integral is estimated from $N$ samples $t_1<\dots<t_N$ with spacings $\delta_i=t_{i+1}-t_i$, using the quadrature rule from Max's 1995 review of volume rendering. Within each interval the density and colour are treated as constant:

$$
\hat C(\mathbf r)=\sum_{i=1}^{N}T_i\,\big(1-e^{-\sigma_i\delta_i}\big)\,\mathbf c_i,\qquad T_i=\exp\!\Big(-\sum_{j<i}\sigma_j\delta_j\Big)=\prod_{j<i}(1-\alpha_j),\qquad \alpha_i=1-e^{-\sigma_i\delta_i}
$$

- $\alpha_i$: opacity of interval $i$: the chance the ray stops inside it, given it got there
- $w_i=T_i\alpha_i$: compositing weight of sample $i$; $\sum_iw_i=1-T_{N+1}\le1$ and the rest is background
- $\delta_i$: distance to the next sample; the reference code sets the last one to $10^{10}$ so the far end is opaque
- $\sum_iw_it_i$: expected termination depth, a free depth map

This is front-to-back **alpha compositing** with $\alpha_i$ derived from density. Every operation is differentiable in $\sigma_i$ and $\mathbf c_i$, so the gradient of a pixel error flows back into the MLP at every sample. The training loss is the squared error between rendered and photographed pixels over a batch $\mathcal R$ of rays:

$$
\mathcal L=\sum_{\mathbf r\in\mathcal R}\Big[\big\|\hat C_c(\mathbf r)-C(\mathbf r)\big\|_2^2+\big\|\hat C_f(\mathbf r)-C(\mathbf r)\big\|_2^2\Big]
$$

- $C(\mathbf r)$: ground-truth colour of the pixel the ray passes through
- $\hat C_c,\ \hat C_f$: renderings by the coarse and fine networks (next sections); both are supervised

## Positional encoding and spectral bias

An MLP fed raw $(x,y,z,\theta,\phi)$ renders blurry images. Neural networks are biased toward low-frequency functions (Rahaman et al. 2019, "spectral bias"), and a scene has sharp edges and fine texture. NeRF passes each coordinate through a fixed map into sines and cosines of geometrically increasing frequency:

$$
\gamma(p)=\big(\sin(2^0\pi p),\ \cos(2^0\pi p),\ \dots,\ \sin(2^{L-1}\pi p),\ \cos(2^{L-1}\pi p)\big),\qquad L=10\ \text{for }\mathbf x,\quad L=4\ \text{for }\mathbf d
$$

- $p$: one coordinate, normalised to $[-1,1]$; $\gamma$ is applied to each of the 3 components separately
- $L$: number of octaves: $\gamma(\mathbf x)\in\mathbb R^{60}$, $\gamma(\mathbf d)\in\mathbb R^{24}$ (the reference code also appends the raw input, giving 63 and 27)
- $2^{L-1}\pi$: highest frequency; it sets the finest detail the field can easily express

This is the same idea as the sinusoidal position code of the Transformer, applied to continuous coordinates. Tancik et al. (NeurIPS 2020, "Fourier Features Let Networks Learn High Frequency Functions") explained why it works. Through the neural tangent kernel, a Fourier feature map turns the MLP's kernel into a stationary one whose bandwidth is set by the frequencies. Too few octaves gives blur. Too many gives noisy, aliased fits between training views.

## Network and view dependence

The MLP has 8 fully connected ReLU layers of 256 channels. $\gamma(\mathbf x)$ is concatenated back in at the fifth layer (a skip connection). The trunk outputs $\sigma$ (through a ReLU) and a 256-d feature. The feature is concatenated with $\gamma(\mathbf d)$ and passed through one 128-channel layer to a sigmoid RGB. Because $\sigma$ is computed **before** the direction enters, the geometry is forced to be multi-view consistent. Only colour can vary with view, which is how NeRF reproduces specular reflections. In the paper's ablation on the synthetic scenes, removing positional encoding cost 2.2 dB PSNR (31.01 → 28.77) and removing view dependence 3.4 dB (→ 27.66).

## Hierarchical sampling: 64 coarse + 128 fine

Evaluating the MLP densely along every ray wastes most queries on empty space and occluded regions. NeRF trains two networks. The **coarse** network is evaluated at $N_c=64$ stratified points. Split $[t_n,t_f]$ into $N_c$ equal bins and draw one uniform sample in each, so that over training the network sees continuous positions, not a fixed grid. Its weights, normalised to $\hat w_i=w_i/\sum_jw_j$, define a piecewise-constant PDF along the ray. $N_f=128$ more points are drawn from it by inverse-transform sampling. The **fine** network is then evaluated at all $N_c+N_f=192$ points, and $\hat C_f$ is the final rendering. The fine samples land where the coarse pass found surfaces. This is importance sampling of the rendering integral.

**Setup in the paper**

- **Data:** Diffuse Synthetic 360°, Realistic Synthetic 360° (8 Blender scenes, 100 training views at 800×800), and Real Forward-Facing (8 LLFF scenes, 20–62 phone images, poses from COLMAP)
- **Network:** 8×256 ReLU MLP, skip at layer 5, one 128-channel view-dependent layer; $L=10$ for $\mathbf x$, $L=4$ for $\mathbf d$
- **Sampling:** $N_c=64$ coarse + $N_f=128$ fine samples per ray; coarse and fine networks both trained
- **Optimisation:** batch of 4096 rays, Adam with learning rate $5\times10^{-4}$ decaying exponentially to $5\times10^{-5}$
- **Cost:** 100k–300k iterations, about 1–2 days per scene on one NVIDIA V100
- **Storage:** about 5 MB of network weights per scene, versus over 15 GB of sampled volumes for LLFF on the synthetic scenes

> Figure: A 2-D scene seen from above: a fog disc (σ = 2.5), two solid discs (σ = 30) and a thin rod 0.07 wide (σ = 60). The camera at o casts one ray, rendered with the NeRF quadrature between near 0.1 and far 2.6. The reference uses 4000 evenly spaced samples. 'error' is the L1 distance between the estimated and reference colour weights of the four objects plus background. At the default 18° the ray hits only the rod. The reference gives it weight 0.984 (opacity 0.988, depth 1.186). Eight stratified coarse samples step over it: opacity 0.084, background weight 0.916, error 1.922. Sixteen coarse samples are still wrong (1.502), and 32 get it right (0.008). Now add Nf = 16 fine samples drawn from the coarse weights. The coarse pass gave the rod only 0.025 of weight, but that is enough to put a PDF peak there. The error drops to 0.012 at 8 + (8 + 16) = 32 MLP queries. Press resample twice (seed 3): the coarse pass sees nothing at the rod, the fine samples go elsewhere, and the error is 1.968. Over 100 jitter seeds at 18°, the run succeeds (error < 0.05) 75 times for 8 + 16, 79 times for 32 uniform samples, and 95 times for 16 + 16. At 0° the ray crosses the fog and clips both solids. There 8 + 16 is worse than 32 uniform samples (mean error 0.172 vs 0.038): the fine samples cluster at the surfaces, while the fog is still integrated with 8 coarse points. With the paper's 64 + 128, every readout is within 0.01 of the reference.

## Worked example: four samples on one ray

One grey channel. Samples at $t=1.0,1.5,2.0,2.5$ with $\delta_i=0.5$ for all four (no infinite last interval here), densities $\sigma=(0,1,4,2)$ and colours $c=(0.2,0.9,0.5,0.1)$.

Volume-rendering quadrature, computed in Python

| i | tᵢ | σᵢδᵢ | Tᵢ | αᵢ = 1 − e^(−σᵢδᵢ) | wᵢ = Tᵢαᵢ | wᵢcᵢ | ŵᵢ (fine PDF) |
| --- | --- | --- | --- | --- | --- | --- | --- |
| 1 | 1.0 | 0.0 | 1.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 |
| 2 | 1.5 | 0.5 | 1.0000 | 0.3935 | 0.3935 | 0.3541 | 0.4057 |
| 3 | 2.0 | 2.0 | 0.6065 | 0.8647 | 0.5244 | 0.2622 | 0.5408 |
| 4 | 2.5 | 1.0 | 0.0821 | 0.6321 | 0.0519 | 0.0052 | 0.0535 |

**Reading the table**

1. Transmittance accumulates optical depth: $T_3=e^{-(0+0.5)}=0.6065$, $T_4=e^{-(0+0.5+2)}=0.0821$. Sample 4 is mostly hidden behind sample 3.
2. Pixel value $\hat C=\sum_iw_ic_i=0.3541+0.2622+0.0052=0.6215$. Opacity $\sum_iw_i=0.9698$, so the background shows through with weight $T_5=0.0302$.
3. Expected depth $\sum_iw_it_i=1.7688$, or $1.8239$ after dividing by the opacity. The highest weight is at $t=2.0$, where $\sigma$ peaks.
4. For hierarchical sampling, the normalised weights $\hat w=(0,\,0.406,\,0.541,\,0.054)$ are the PDF. About 95% of the fine samples go to the bins around $t=1.5$ and $t=2.0$, and none to $t=1.0$.
5. Positional encoding of one coordinate $p=0.3$ with $L=3$: $(\sin,\cos)$ pairs $(0.809,0.588)$, $(0.951,-0.309)$, $(-0.588,-0.809)$ for frequencies $\pi,2\pi,4\pi$. A shift of 0.1 in $p$ rotates the third pair by $0.4\pi$, so the last octaves are what resolve fine detail.

```python
import torch, torch.nn as nn, torch.nn.functional as F

def posenc(p, L):
    # p (..., 3) -> (..., 3 + 6L): [p, sin(2^k pi p), cos(2^k pi p)] for k < L
    freqs = (2.0 ** torch.arange(L)) * torch.pi                      # (L,)
    ang = p[..., None, :] * freqs[:, None]                           # (..., L, 3)
    return torch.cat([p, ang.sin().flatten(-2), ang.cos().flatten(-2)], -1)

class NeRF(nn.Module):
    def __init__(self, Lx=10, Ld=4, W=256):
        super().__init__()
        self.Lx, self.Ld = Lx, Ld
        dx, dd = 3 + 6 * Lx, 3 + 6 * Ld                               # 63, 27
        self.l1 = nn.ModuleList([nn.Linear(dx, W)] + [nn.Linear(W, W) for _ in range(4)])
        self.l2 = nn.ModuleList([nn.Linear(W + dx, W)] + [nn.Linear(W, W) for _ in range(2)])  # skip into layer 5
        self.sigma = nn.Linear(W, 1)
        self.feat = nn.Linear(W, W)
        self.rgb = nn.Sequential(nn.Linear(W + dd, W // 2), nn.ReLU(), nn.Linear(W // 2, 3), nn.Sigmoid())

    def forward(self, x, d):
        gx, gd = posenc(x, self.Lx), posenc(d, self.Ld)
        h = gx
        for l in self.l1: h = F.relu(l(h))
        h = torch.cat([h, gx], -1)
        for l in self.l2: h = F.relu(l(h))
        sigma = F.relu(self.sigma(h)).squeeze(-1)                     # density depends on x only
        rgb = self.rgb(torch.cat([self.feat(h), gd], -1))             # colour also sees the view direction
        return rgb, sigma

def composite(rgb, sigma, t):
    # rgb (R, N, 3), sigma (R, N), t (R, N) sorted sample depths
    delta = torch.cat([t[:, 1:] - t[:, :-1], torch.full_like(t[:, :1], 1e10)], -1)
    alpha = 1 - torch.exp(-sigma * delta)
    T = torch.cumprod(torch.cat([torch.ones_like(alpha[:, :1]), 1 - alpha + 1e-10], -1), -1)[:, :-1]
    w = T * alpha                                                     # (R, N)
    return (w[..., None] * rgb).sum(1), w

def sample_pdf(t, w, n):
    # inverse-transform sampling from the piecewise-constant PDF of the coarse weights
    mids = 0.5 * (t[:, 1:] + t[:, :-1])                               # (R, N-1) bin edges
    pdf = (w[:, 1:-1] + 1e-5); pdf = pdf / pdf.sum(-1, keepdim=True)
    cdf = torch.cat([torch.zeros_like(pdf[:, :1]), pdf.cumsum(-1)], -1)  # (R, N-1)
    u = torch.rand(t.shape[0], n)
    i = torch.searchsorted(cdf, u, right=True).clamp(1, cdf.shape[-1] - 1)
    c0, c1 = cdf.gather(-1, i - 1), cdf.gather(-1, i)
    b0, b1 = mids.gather(-1, i - 1), mids.gather(-1, i)
    return b0 + (u - c0) / (c1 - c0).clamp_min(1e-5) * (b1 - b0)

def render(coarse, fine, o, d, near=2.0, far=6.0, Nc=64, Nf=128):
    R = o.shape[0]
    t = near + (far - near) * (torch.arange(Nc) + torch.rand(R, Nc)) / Nc   # stratified
    x = o[:, None] + t[..., None] * d[:, None]
    rgb_c, w = composite(*coarse(x, d[:, None].expand_as(x)), t)
    t_f = torch.sort(torch.cat([t, sample_pdf(t, w.detach(), Nf)], -1), -1).values  # Nc + Nf
    x = o[:, None] + t_f[..., None] * d[:, None]
    rgb_f, _ = composite(*fine(x, d[:, None].expand_as(x)), t_f)
    return rgb_c, rgb_f

coarse, fine = NeRF(), NeRF()
print(sum(p.numel() for p in coarse.parameters()))           # 595844 parameters per network
o = torch.zeros(1024, 3); d = F.normalize(torch.randn(1024, 3), dim=-1)
target = torch.rand(1024, 3)
rgb_c, rgb_f = render(coarse, fine, o, d)
loss = ((rgb_c - target) ** 2).sum(-1).mean() + ((rgb_f - target) ** 2).sum(-1).mean()
loss.backward()
print(rgb_c.shape, rgb_f.shape, posenc(o, 10).shape, posenc(d, 4).shape)
# torch.Size([1024, 3]) torch.Size([1024, 3]) torch.Size([1024, 63]) torch.Size([1024, 27])
```

NeRF MLP, stratified + hierarchical sampling, volume rendering and the two-term loss (runs on CPU)

## Results

Mildenhall et al. (2020), Table 1: PSNR ↑ / SSIM ↑ / LPIPS ↓

| method | Realistic Synthetic 360° | Real Forward-Facing (LLFF) |
| --- | --- | --- |
| SRN | 22.26 / 0.846 / 0.170 | 22.84 / 0.668 / 0.378 |
| Neural Volumes | 26.05 / 0.893 / 0.160 | — |
| LLFF | 24.88 / 0.911 / 0.114 | 24.13 / 0.798 / 0.212 |
| NeRF | 31.01 / 0.947 / 0.081 | 26.50 / 0.811 / 0.250 |

> **Hierarchical sampling only refines what the coarse pass saw:** The fine PDF is built from coarse weights. A thin surface that falls between all $N_c$ stratified samples gets no weight, so no fine sample goes there. That ray then renders background (seed 3 at 18° in the figure). Stratified jitter fixes this **on average over training**, because each iteration probes new positions. Do not drop the jitter at train time, and do not shrink $N_c$ too far to save compute. In the toy, 8 coarse + 16 fine samples did worse than 32 uniform samples on a foggy ray. The win is on rays that are mostly empty space with a few sharp surfaces, which is the common case in real scenes.

> **Poses, scale and floaters:** NeRF assumes exact camera poses, usually from COLMAP structure-from-motion, and a static scene with constant lighting. Pose errors or moving objects produce blur and semi-transparent "floaters" that explain one view while spoiling others. Unbounded outdoor scenes need a reparametrisation of far space (NDC rays for forward-facing captures in the paper, scene contraction in Mip-NeRF 360). Normalise the scene so that $\mathbf x$ lies in the range the positional encoding expects.

> **Rendering is the generative model's likelihood:** NeRF has no encoder, no latent prior and no adversary. The only learning signal is a differentiable renderer compared with photographs. Swap the photographs for another supervisory signal and the same machinery generates 3-D content. **DreamFusion** (Poole, Jain, Barron and Mildenhall, 2022) optimises a NeRF from a text prompt alone. It renders random views and pushes them toward images a frozen text-to-image diffusion model finds likely, via **score distillation sampling**: $\nabla_\theta\mathcal L_{\text{SDS}}=\mathbb E_{t,\epsilon}\big[w(t)(\hat\epsilon_\phi(\mathbf z_t;y,t)-\epsilon)\,\partial\mathbf x/\partial\theta\big]$, see **Diffusion models**. GAN-based 3-D generators (GRAF 2020, pi-GAN 2021, EG3D 2022) instead put a radiance field inside the generator of a **Generative adversarial network (GAN)**, and train it so random renderings fool a 2-D discriminator.

> **What came next:** **Instant-NGP** (Müller, Evans, Schied and Keller, SIGGRAPH 2022) replaces most of the MLP and the fixed sinusoids with a learned **multiresolution hash encoding**: trainable feature tables at 16 grid resolutions, indexed by a spatial hash and followed by a tiny MLP. Training drops from days to seconds or minutes. **mip-NeRF** (Barron et al., ICCV 2021) renders cones instead of rays. It encodes each conical frustum by an integrated positional encoding, which removes aliasing across scales. Mip-NeRF 360 (CVPR 2022) extends it to unbounded scenes. **3D Gaussian splatting** keeps NeRF's compositing equation but drops the network and ray-marching for explicit primitives that are rasterised in real time.

**Strengths**

- Continuous, compact scene: about 5 MB of weights instead of gigabytes of voxels or MPIs
- Photorealistic novel views with view-dependent effects; geometry and depth come for free from σ
- Fully differentiable, so the renderer can sit inside larger learning systems (pose refinement, text-to-3D, 3-D-aware GANs)
- No 3-D supervision: only posed 2-D images

**Limitations**

- Slow: 192 MLP evaluations per pixel, far from real-time rendering, and 1–2 days of training per scene in the original
- One network per scene, no generalisation across scenes without extra machinery
- Needs accurate poses, a static scene and fixed lighting; prone to floaters with sparse views
- Aliasing when rendering at resolutions or distances unlike the training views (addressed by mip-NeRF)

**Question:** Why does the NeRF MLP output σ before the viewing direction d is fed in?

- A. So that density, and thus geometry, is the same from every viewpoint while colour can still vary with view
- B. Because σ must be in [0, 1] and the direction would break that
- C. To halve the number of parameters
- D. Because the positional encoding of d is too high-dimensional

**Answer:** A. So that density, and thus geometry, is the same from every viewpoint while colour can still vary with view — If $\sigma$ depended on $\mathbf d$, the network could explain each training image with different geometry, and new views would be inconsistent. Restricting view dependence to $\mathbf c(\mathbf x,\mathbf d)$ enforces multi-view consistency, while still modelling specular highlights.

**Question:** Two samples on a ray have σδ = 0.5 and then σδ = 2.0. What is the compositing weight of the second sample?

- A. $1-e^{-2}=0.865$
- B. $e^{-0.5}(1-e^{-2})=0.524$
- C. $e^{-2.5}=0.082$
- D. $(1-e^{-0.5})(1-e^{-2})=0.340$

**Answer:** B. $e^{-0.5}(1-e^{-2})=0.524$ — $w_2=T_2\alpha_2$ with $T_2=e^{-0.5}=0.6065$ (survive the first interval) and $\alpha_2=1-e^{-2}=0.8647$, giving $0.5244$, as in the worked example. Option A ignores occlusion by the first sample. Option C is the transmittance after both.

**Key points**

- NeRF rendering: $\hat C=\sum_iT_i(1-e^{-\sigma_i\delta_i})\mathbf c_i$ with $T_i=\exp(-\sum_{j<i}\sigma_j\delta_j)$; the MLP $F(\mathbf x,\mathbf d)\to(\mathbf c,\sigma)$ is fitted per scene with $\sum_{\mathbf r}\|\hat C(\mathbf r)-C(\mathbf r)\|^2$.
- Positional encoding $\gamma(p)=(\sin 2^k\pi p,\cos 2^k\pi p)_{k<L}$ with $L=10$ for position and $L=4$ for direction overcomes the MLP's spectral bias toward smooth functions.
- Hierarchical sampling: 64 stratified coarse samples; their normalised weights form a PDF for 128 fine samples; the fine network renders all 192.
- $\sigma$ depends on $\mathbf x$ only and $\mathbf c$ on $(\mathbf x,\mathbf d)$, which gives consistent geometry and view-dependent appearance.
- Realistic Synthetic 360°: 31.01 dB PSNR versus 26.05 for the best prior method; cost about 1–2 days of training per scene on a V100.
- Successors: Instant-NGP hash grids (minutes), mip-NeRF cones (anti-aliasing), DreamFusion's score distillation (text-to-3D), 3D Gaussian splatting (real time).

---

From MLFoundry (https://www.mlfoundry.in). The web page adds interactive figures and diagrams.
