# 3D reconstruction

> 3D reconstruction recovers geometry from images: structure-from-motion finds poses and sparse points by triangulation and bundle adjustment, then multi-view stereo (PatchMatch, MVSNet) estimates dense depth.

Source: https://www.mlfoundry.in/learn/3d-reconstruction  
Section: Deep Learning › Deep Learning for Computer Vision  
Level: advanced · 8 min read · updated 2026-10-15

**3D reconstruction:** recovers the 3D shape of a scene, and usually the camera poses, from 2D images. The standard multi-view pipeline is **structure from motion** (SfM), which estimates cameras and a sparse point cloud, followed by **multi-view stereo** (MVS), which estimates dense depth. The reference open-source system is COLMAP: "Structure-from-Motion Revisited" (Schönberger and Frahm, CVPR 2016) and "Pixelwise View Selection for Unstructured Multi-View Stereo" (Schönberger, Zheng, Frahm and Pollefeys, ECCV 2016). Deep learning replaces individual stages, for example MVSNet (Yao, Luo, Li, Fang and Quan, ECCV 2018) for dense depth.

Input: a set of photos of a static scene, often with unknown poses and sometimes unknown intrinsics. Output: camera parameters plus a point cloud, mesh, implicit surface or radiance field. The difficulty is that projection discards depth. Depth comes back only through **parallax**: the same point seen from two or more positions.

Each camera is a pinhole. A world point $X$ projects to pixel $x$ through the intrinsics $K$ and the pose $(R,t)$. SfM inverts this for all cameras and points at once, by minimising the **reprojection error** in bundle adjustment (Triggs et al., 2000):

$$
\lambda\begin{bmatrix}x\\1\end{bmatrix}=K\,[R\mid t]\begin{bmatrix}X\\1\end{bmatrix},\qquad \min_{\{R_i,t_i\},\{X_j\}}\ \sum_{(i,j)\in\mathcal O}\rho\Big(\big\|x_{ij}-\pi\big(K_i(R_iX_j+t_i)\big)\big\|^2\Big)
$$

- $K$: intrinsics: focal lengths $f_x,f_y$ in pixels and principal point $(c_x,c_y)$
- $\pi$: perspective division $(a,b,c)\mapsto(a/c,\,b/c)$; $\lambda$ is the depth of the point
- $\mathcal O$: observations: point $j$ was matched in image $i$ at pixel $x_{ij}$
- $\rho$: robust loss (Huber, Cauchy) so that wrong matches do not dominate

## Structure from motion, then multi-view stereo

SfM starts with local features: SIFT keypoints with 128-D descriptors (Lowe, 2004), matched by nearest neighbour with Lowe's ratio test. Matches between two views must satisfy the epipolar constraint $\hat x_2^\top E\,\hat x_1=0$ with $E=[t]_\times R$ in normalised coordinates. RANSAC with the five-point algorithm (Nistér, 2004) estimates $E$ and rejects outliers. Incremental SfM then grows the model one image at a time: register the new camera from 2D–3D matches (PnP), triangulate new points, and run bundle adjustment.

> Figure: The COLMAP-style pipeline. Steps 1–5 are structure from motion and output poses K, R, t plus a sparse cloud. Steps 6–9 are multi-view stereo: with poses fixed, estimate a depth for every pixel of every view, fuse the depth maps that agree, then build a surface (or train a NeRF or 3D Gaussian splatting model on the posed images). The lower row expands step 6 as MVSNet: features at quarter resolution with 32 channels are warped onto D fronto-parallel planes of the reference camera, combined by their variance across views, regularised by a 3D CNN and turned into depth by a softmax-weighted average.

MVS assumes the poses are known. Classical MVS such as COLMAP's PatchMatch stereo searches, for every pixel, the depth and normal whose patch looks most alike in the other views (normalised cross-correlation). It propagates good hypotheses to neighbouring pixels. Depth maps are then fused into a point cloud by keeping only depths that are consistent across views.

Depth accuracy follows from triangulation geometry. For two parallel cameras with baseline $b$, depth comes from the disparity $d=u_1-u_2$, and its error grows with the square of the depth:

$$
Z=\frac{f\,b}{d},\qquad \delta Z\approx\frac{Z^2}{f\,b}\,\delta d
$$

- $f$: focal length in pixels
- $b$: baseline: distance between the camera centres, in metres
- $\delta d$: disparity error; with independent noise $\sigma$ on each pixel coordinate, $\delta d=\sigma\sqrt2$

## Learned multi-view stereo

MVSNet keeps the geometry and learns the matching. A shared 2D CNN extracts 32-channel features at a quarter of the input resolution. Each source feature map is warped into the reference view by the homography of each of $D$ depth planes. The warped volumes are merged by their variance, which works for any number of views. A 3D CNN regularises the cost volume, a softmax along depth gives $P(d)$, and the depth is its expectation:

$$
C=\frac1N\sum_{i=1}^{N}\big(V_i-\bar V\big)^2,\qquad \hat d(p)=\sum_{j=1}^{D}d_j\,P_j(p),\qquad P(p)=\operatorname{softmax}_j\big(\mathrm{CNN_{3D}}(C)_j(p)\big)
$$

- $V_i$: features of view $i$ warped onto the $D$ planes, shape $D\times\tfrac H4\times\tfrac W4\times32$
- $C$: low variance at a plane means the views agree, so the true depth is likely there
- $\hat d$: soft argmin: differentiable, and sub-plane accurate; trained with an L1 loss to ground-truth depth

The cost volume costs memory proportional to $D\cdot H\cdot W$. Later methods (R-MVSNet with recurrent regularisation, cascade cost volumes with coarse-to-fine depth ranges) reduce it. A different line drops the explicit pipeline. DUSt3R (Wang et al., CVPR 2024) regresses a **pointmap**, a 3D point per pixel in the first camera's frame, directly from an uncalibrated image pair with a transformer. Poses and intrinsics are then read off the pointmaps.

Scene representations (memory figures computed for float32)

| Representation | What is stored | Strength | Weakness |
| --- | --- | --- | --- |
| Point cloud | xyz (+ colour, normal) per point | direct output of MVS and SfM | no surface, holes |
| Voxel grid | occupancy or TSDF per cell | simple 3D CNNs (3D-R2N2, Choy et al. 2016) | cubic memory: 256³ cells = 16.8M = 67 MB |
| Mesh | vertices + triangles | compact, renders everywhere (Pixel2Mesh deforms an ellipsoid, Wang et al. 2018) | fixed topology is hard to learn |
| Implicit surface | network f(x) = signed distance or occupancy | continuous resolution (DeepSDF, Occupancy Networks, both CVPR 2019) | needs marching cubes to extract a mesh |
| Radiance field | density + view-dependent colour (MLP or Gaussians) | photorealistic novel views | geometry is a by-product, can be noisy |

Radiance fields are covered in **Neural radiance fields (NeRF)** and **3D Gaussian splatting**. Both are optimised per scene, and both normally take their camera poses (and, for 3DGS, the initial points) from COLMAP. Monocular depth networks are covered in **Depth estimation**.

**Benchmarks and metrics**

- **DTU:** indoor objects captured by a robot-arm camera with structured-light ground truth (Aanæs et al., 2016). Reports accuracy (prediction → ground truth distance), completeness (ground truth → prediction) and their mean, in mm.
- **Tanks and Temples:** real indoor/outdoor scenes with laser-scanned ground truth (Knapitsch, Park, Zhou and Koltun, SIGGRAPH 2017). Reports the F-score of precision and recall at a distance threshold τ.
- **Learned matching:** SuperPoint keypoints (DeTone et al., 2018) with the SuperGlue graph-network matcher (Sarlin et al., CVPR 2020) are a common drop-in for SIFT + ratio test inside SfM.

> Figure: Linear (DLT) triangulation of one point, repeated 200 times with Gaussian pixel noise (top-down slice, f = 500 px). Defaults: two cameras 0.5 m apart, point at 5 m, σ = 0.5 px. The disparity is 50 px and the triangulation angle 5.69°. Depth RMS error is 7.5 cm, close to the first-order prediction Z²σ√2/(fb) = 7.1 cm, while lateral error is only 0.7 cm, so the cloud is a thin vertical streak. Move the point to 10 m: the error becomes 30.3 cm (about 4×). A 2 m baseline at 10 m brings it back to 7.5 cm. Five cameras on the same 0.5 m baseline at 5 m only reach 6.5 cm: spread of viewpoints matters more than count.

## Worked example: stereo triangulation

f = 500 px, cameras at x = ±0.25 m (b = 0.5 m), true point X = 0.4 m, Z = 5 m (computed in Python)

| Quantity | Formula | Value |
| --- | --- | --- |
| u₁ (left camera) | f·(X − c₁)/Z = 500·0.65/5 | 65 px |
| u₂ (right camera) | f·(X − c₂)/Z = 500·0.15/5 | 15 px |
| disparity d | u₁ − u₂ | 50 px |
| depth Z | f·b/d = 250/50 | 5.000 m |
| lateral X | c₁ + Z·u₁/f = −0.25 + 0.65 | 0.400 m |
| Z if d is off by 1 px | 250/49 | 5.102 m (error 10.2 cm) |
| same at Z = 10 m (d = 25 px) | 250/24 | 10.417 m (error 41.7 cm) |

The linear rule $\delta Z\approx Z^2\delta d/(fb)$ gives 10 cm and 40 cm for these two cases. A one-pixel error costs four times more depth at twice the distance. This is why outdoor scenes need wide baselines, and why distant points in SfM have large uncertainty along the viewing ray.

```python
from pathlib import Path
import pycolmap

images, out = Path("images"), Path("out")
out.mkdir(exist_ok=True)
db = out / "database.db"

pycolmap.extract_features(db, images)            # SIFT keypoints + descriptors
pycolmap.match_exhaustive(db)                   # all pairs, ratio test, RANSAC on E/F
maps = pycolmap.incremental_mapping(db, images, out)   # PnP + triangulation + bundle adjustment
rec = maps[0]
rec.write(out)
print(rec.summary())                            # registered images, points, mean reprojection error

mvs = out / "mvs"
pycolmap.undistort_images(mvs, out, images)    # (output, sparse model, images)
pycolmap.patch_match_stereo(mvs)                # per-view depth and normal maps
pycolmap.stereo_fusion(mvs / "dense.ply", mvs)  # fuse consistent depths into a dense cloud
```

Sparse SfM then dense MVS with pycolmap, COLMAP's Python bindings (not run here; PatchMatch stereo needs a CUDA build)

> **Scale and degenerate motion:** From images alone, reconstruction is defined only up to a similarity transform: scaling the scene and all translations by the same factor leaves every projection unchanged. Metric scale needs a known baseline, a calibration object, GPS or an IMU. Pure rotation about the camera centre gives no parallax at all, so nothing can be triangulated. Repetitive textures and reflective or textureless surfaces break the matching, not the geometry.

> **Learning slots into a geometric skeleton:** Most successful learned systems keep the projective geometry and learn only the ambiguous part: features and matching (SuperPoint, SuperGlue), cost-volume regularisation (MVSNet), or appearance (NeRF, 3DGS on COLMAP poses). Methods such as DUSt3R that regress geometry directly are strong where classical matching fails, for example with few views and little overlap.

**Question:** A stereo rig measures a point at 4 m with a depth error of 2 cm. With the same cameras and matching noise, what error do you expect at 12 m?

- A. 6 cm
- B. 12 cm
- C. 18 cm
- D. 36 cm

**Answer:** C. 18 cm — $\delta Z\propto Z^2$ at fixed $f$, $b$ and $\delta d$. Tripling the depth multiplies the error by 9: $2\times9=18$ cm. Doubling the baseline would halve it.

**Question:** Why does MVSNet merge the warped feature volumes with a variance instead of concatenating them?

- A. Variance works for any number of source views and is low exactly where the views agree
- B. Concatenation cannot be differentiated through the homography warp
- C. Variance removes the need for camera poses
- D. The 3D CNN only accepts single-channel input

**Answer:** A. Variance works for any number of source views and is low exactly where the views agree — The variance has the same shape whatever N is, so one network handles 3, 5 or 10 views. At the correct depth plane all views see the same surface point, their features agree and the variance is small; the 3D CNN and the softmax turn this into a depth probability.

**Key points**

- SfM: features → matching with RANSAC on the essential matrix → incremental PnP registration, triangulation and bundle adjustment of the reprojection error. COLMAP is the standard implementation.
- MVS: with poses fixed, estimate per-pixel depth (PatchMatch photo-consistency, or a learned cost volume) and fuse the consistent depth maps.
- Stereo depth $Z=fb/d$ and its error $\delta Z\approx Z^2\delta d/(fb)$: error grows with the square of distance and falls with baseline.
- MVSNet: homography warping onto D depth planes, variance cost volume, 3D CNN, soft argmin depth.
- Representations trade memory and topology: points, voxels (cubic memory), meshes, implicit SDF/occupancy, radiance fields (NeRF, 3DGS on COLMAP poses).

---

From MLFoundry (https://www.mlfoundry.in). The web page adds interactive figures and diagrams.
