3D reconstruction

3D reconstruction recovers geometry from images: structure-from-motion finds poses and sparse points by triangulation and bundle adjustment, then multi-view stereo (PatchMatch, MVSNet) estimates dense depth.

8 min read802 wordsadvanced0 viewsupdated 2026-10-15

Definition

3D reconstruction recovers the 3D shape of a scene, and usually the camera poses, from 2D images. The standard multi-view pipeline is structure from motion (SfM), which estimates cameras and a sparse point cloud, followed by multi-view stereo (MVS), which estimates dense depth. The reference open-source system is COLMAP: "Structure-from-Motion Revisited" (Schönberger and Frahm, CVPR 2016) and "Pixelwise View Selection for Unstructured Multi-View Stereo" (Schönberger, Zheng, Frahm and Pollefeys, ECCV 2016). Deep learning replaces individual stages, for example MVSNet (Yao, Luo, Li, Fang and Quan, ECCV 2018) for dense depth.

Input: a set of photos of a static scene, often with unknown poses and sometimes unknown intrinsics. Output: camera parameters plus a point cloud, mesh, implicit surface or radiance field. The difficulty is that projection discards depth. Depth comes back only through parallax: the same point seen from two or more positions.

Each camera is a pinhole. A world point XX projects to pixel xx through the intrinsics KK and the pose (R,t)(R,t). SfM inverts this for all cameras and points at once, by minimising the reprojection error in bundle adjustment (Triggs et al., 2000):

λ[x1]=K [R∣t][X1],min⁡{Ri,ti},{Xj} ∑(i,j)∈Oρ(∥xij−π(Ki(RiXj+ti))∥2)\lambda\begin{bmatrix}x\\1\end{bmatrix}=K\,[R\mid t]\begin{bmatrix}X\\1\end{bmatrix},\qquad \min_{\{R_i,t_i\},\{X_j\}}\ \sum_{(i,j)\in\mathcal O}\rho\Big(\big\|x_{ij}-\pi\big(K_i(R_iX_j+t_i)\big)\big\|^2\Big)
KK
intrinsics: focal lengths fx,fyf_x,f_y in pixels and principal point (cx,cy)(c_x,c_y)
π\pi
perspective division (a,b,c)↦(a/c, b/c)(a,b,c)\mapsto(a/c,\,b/c); λ\lambda is the depth of the point
O\mathcal O
observations: point jj was matched in image ii at pixel xijx_{ij}
ρ\rho
robust loss (Huber, Cauchy) so that wrong matches do not dominate

Structure from motion, then multi-view stereo

SfM starts with local features: SIFT keypoints with 128-D descriptors (Lowe, 2004), matched by nearest neighbour with Lowe's ratio test. Matches between two views must satisfy the epipolar constraint x^2⊤E x^1=0\hat x_2^\top E\,\hat x_1=0 with E=[t]×RE=[t]_\times R in normalised coordinates. RANSAC with the five-point algorithm (Nistér, 2004) estimates EE and rejects outliers. Incremental SfM then grows the model one image at a time: register the new camera from 2D–3D matches (PnP), triangulate new points, and run bundle adjustment.

structure from motion → multi-view stereo (COLMAP-style pipeline)1 · imagesN photos,unknown poses2 · featuresSIFT keypoints,128-D descriptors3 · matchNN + ratio test,RANSAC on E / F4 · incrementalPnP register,triangulate, BA5 · sparse modelK, R, t per view+ sparse pointsposes fixed; now estimate depth for every pixel6 · MVS depthone depth map perview: PatchMatchor MVSNet7 · fusionkeep depths thatagree across views8 · dense cloudpoints + normals+ colours9 · surfacePoisson mesh, TSDF+ marching cubes,or NeRF / 3DGSstep 6, learned: MVSNet (Yao et al., 2018) for one reference viewsrc I₂I₀ + N−1 viewsknown K, R, t2D CNNshared weights¼H × ¼W × 32warp onto D planes d_jby homography H_i(d_j)C = Var_i(V_i): D×¼H×¼W×323D CNNregularisesoftmax over d→ P(d) per pixeld₁ … d_Ddepth = Σ dⱼ·P(dⱼ)(soft argmin)loss: L1 between predicted and ground-truth depth (DTU) · output: one depth map, then fusion (step 7)classical step 6 instead scores photo-consistency (NCC) of patches and propagates good depthsBA = bundle adjustment: jointly refine all poses and points to minimise reprojection error
fig.The COLMAP-style pipeline. Steps 1–5 are structure from motion and output poses K, R, t plus a sparse cloud. Steps 6–9 are multi-view stereo: with poses fixed, estimate a depth for every pixel of every view, fuse the depth maps that agree, then build a surface (or train a NeRF or 3D Gaussian splatting model on the posed images). The lower row expands step 6 as MVSNet: features at quarter resolution with 32 channels are warped onto D fronto-parallel planes of the reference camera, combined by their variance across views, regularised by a 3D CNN and turned into depth by a softmax-weighted average.

MVS assumes the poses are known. Classical MVS such as COLMAP's PatchMatch stereo searches, for every pixel, the depth and normal whose patch looks most alike in the other views (normalised cross-correlation). It propagates good hypotheses to neighbouring pixels. Depth maps are then fused into a point cloud by keeping only depths that are consistent across views.

Depth accuracy follows from triangulation geometry. For two parallel cameras with baseline bb, depth comes from the disparity d=u1−u2d=u_1-u_2, and its error grows with the square of the depth:

Z=f bd,δZ≈Z2f b δdZ=\frac{f\,b}{d},\qquad \delta Z\approx\frac{Z^2}{f\,b}\,\delta d
ff
focal length in pixels
bb
baseline: distance between the camera centres, in metres
δd\delta d
disparity error; with independent noise σ\sigma on each pixel coordinate, δd=σ2\delta d=\sigma\sqrt2

Learned multi-view stereo

MVSNet keeps the geometry and learns the matching. A shared 2D CNN extracts 32-channel features at a quarter of the input resolution. Each source feature map is warped into the reference view by the homography of each of DD depth planes. The warped volumes are merged by their variance, which works for any number of views. A 3D CNN regularises the cost volume, a softmax along depth gives P(d)P(d), and the depth is its expectation:

C=1N∑i=1N(Vi−Vˉ)2,d^(p)=∑j=1Ddj Pj(p),P(p)=softmax⁡j(CNN3D(C)j(p))C=\frac1N\sum_{i=1}^{N}\big(V_i-\bar V\big)^2,\qquad \hat d(p)=\sum_{j=1}^{D}d_j\,P_j(p),\qquad P(p)=\operatorname{softmax}_j\big(\mathrm{CNN_{3D}}(C)_j(p)\big)
ViV_i
features of view ii warped onto the DD planes, shape D×H4×W4×32D\times\tfrac H4\times\tfrac W4\times32
CC
low variance at a plane means the views agree, so the true depth is likely there
d^\hat d
soft argmin: differentiable, and sub-plane accurate; trained with an L1 loss to ground-truth depth

The cost volume costs memory proportional to D⋅H⋅WD\cdot H\cdot W. Later methods (R-MVSNet with recurrent regularisation, cascade cost volumes with coarse-to-fine depth ranges) reduce it. A different line drops the explicit pipeline. DUSt3R (Wang et al., CVPR 2024) regresses a pointmap, a 3D point per pixel in the first camera's frame, directly from an uncalibrated image pair with a transformer. Poses and intrinsics are then read off the pointmaps.

RepresentationWhat is storedStrengthWeakness
Point cloudxyz (+ colour, normal) per pointdirect output of MVS and SfMno surface, holes
Voxel gridoccupancy or TSDF per cellsimple 3D CNNs (3D-R2N2, Choy et al. 2016)cubic memory: 256³ cells = 16.8M = 67 MB
Meshvertices + trianglescompact, renders everywhere (Pixel2Mesh deforms an ellipsoid, Wang et al. 2018)fixed topology is hard to learn
Implicit surfacenetwork f(x) = signed distance or occupancycontinuous resolution (DeepSDF, Occupancy Networks, both CVPR 2019)needs marching cubes to extract a mesh
Radiance fielddensity + view-dependent colour (MLP or Gaussians)photorealistic novel viewsgeometry is a by-product, can be noisy
Scene representations (memory figures computed for float32)

Radiance fields are covered in Neural radiance fields (NeRF) and 3D Gaussian splatting. Both are optimised per scene, and both normally take their camera poses (and, for 3DGS, the initial points) from COLMAP. Monocular depth networks are covered in Depth estimation.

Benchmarks and metrics

DTU
indoor objects captured by a robot-arm camera with structured-light ground truth (Aanæs et al., 2016). Reports accuracy (prediction → ground truth distance), completeness (ground truth → prediction) and their mean, in mm.
Tanks and Temples
real indoor/outdoor scenes with laser-scanned ground truth (Knapitsch, Park, Zhou and Koltun, SIGGRAPH 2017). Reports the F-score of precision and recall at a distance threshold τ.
Learned matching
SuperPoint keypoints (DeTone et al., 2018) with the SuperGlue graph-network matcher (Sarlin et al., CVPR 2020) are a common drop-in for SIFT + ratio test inside SfM.
top-down · f = 500 px · 2 views, b = 0.5 m0m2m4m6m8m10m12mX = (0.4, 5) mzoom: 200 noisy triangulationsbox = ±0.5 mdepth ↑disparity (outer pair)50.0 pxtriangulation angle5.69°depth RMS error7.5 cm2-view Z²·σ√2 / (f·b)7.1 cmlateral RMS error0.7 cmdepth bias (mean)0.5 cmeach view: f·X − u·Z = f·c (one row of the linear system); noise σ = 0.5 px on every u
views
fig.Linear (DLT) triangulation of one point, repeated 200 times with Gaussian pixel noise (top-down slice, f = 500 px). Defaults: two cameras 0.5 m apart, point at 5 m, σ = 0.5 px. The disparity is 50 px and the triangulation angle 5.69°. Depth RMS error is 7.5 cm, close to the first-order prediction Z²σ√2/(fb) = 7.1 cm, while lateral error is only 0.7 cm, so the cloud is a thin vertical streak. Move the point to 10 m: the error becomes 30.3 cm (about 4×). A 2 m baseline at 10 m brings it back to 7.5 cm. Five cameras on the same 0.5 m baseline at 5 m only reach 6.5 cm: spread of viewpoints matters more than count.

Worked example: stereo triangulation

QuantityFormulaValue
u₁ (left camera)f·(X − c₁)/Z = 500·0.65/565 px
u₂ (right camera)f·(X − c₂)/Z = 500·0.15/515 px
disparity du₁ − u₂50 px
depth Zf·b/d = 250/505.000 m
lateral Xc₁ + Z·u₁/f = −0.25 + 0.650.400 m
Z if d is off by 1 px250/495.102 m (error 10.2 cm)
same at Z = 10 m (d = 25 px)250/2410.417 m (error 41.7 cm)
f = 500 px, cameras at x = ±0.25 m (b = 0.5 m), true point X = 0.4 m, Z = 5 m (computed in Python)

The linear rule δZ≈Z2δd/(fb)\delta Z\approx Z^2\delta d/(fb) gives 10 cm and 40 cm for these two cases. A one-pixel error costs four times more depth at twice the distance. This is why outdoor scenes need wide baselines, and why distant points in SfM have large uncertainty along the viewing ray.

python
1from pathlib import Path
2import pycolmap
3
4images, out = Path("images"), Path("out")
5out.mkdir(exist_ok=True)
6db = out / "database.db"
7
8pycolmap.extract_features(db, images) # SIFT keypoints + descriptors
9pycolmap.match_exhaustive(db) # all pairs, ratio test, RANSAC on E/F
10maps = pycolmap.incremental_mapping(db, images, out) # PnP + triangulation + bundle adjustment
11rec = maps[0]
12rec.write(out)
13print(rec.summary()) # registered images, points, mean reprojection error
14
15mvs = out / "mvs"
16pycolmap.undistort_images(mvs, out, images) # (output, sparse model, images)
17pycolmap.patch_match_stereo(mvs) # per-view depth and normal maps
18pycolmap.stereo_fusion(mvs / "dense.ply", mvs) # fuse consistent depths into a dense cloud
Sparse SfM then dense MVS with pycolmap, COLMAP's Python bindings (not run here; PatchMatch stereo needs a CUDA build)

Check yourself

A stereo rig measures a point at 4 m with a depth error of 2 cm. With the same cameras and matching noise, what error do you expect at 12 m?

Check yourself

Why does MVSNet merge the warped feature volumes with a variance instead of concatenating them?

Remember

  • ✓SfM: features → matching with RANSAC on the essential matrix → incremental PnP registration, triangulation and bundle adjustment of the reprojection error. COLMAP is the standard implementation.
  • ✓MVS: with poses fixed, estimate per-pixel depth (PatchMatch photo-consistency, or a learned cost volume) and fuse the consistent depth maps.
  • ✓Stereo depth Z=fb/dZ=fb/d and its error δZ≈Z2δd/(fb)\delta Z\approx Z^2\delta d/(fb): error grows with the square of distance and falls with baseline.
  • ✓MVSNet: homography warping onto D depth planes, variance cost volume, 3D CNN, soft argmin depth.
  • ✓Representations trade memory and topology: points, voxels (cubic memory), meshes, implicit SDF/occupancy, radiance fields (NeRF, 3DGS on COLMAP poses).
← back to Deep Learning roadmap