Definition
3D reconstruction recovers the 3D shape of a scene, and usually the camera poses, from 2D images. The standard multi-view pipeline is structure from motion (SfM), which estimates cameras and a sparse point cloud, followed by multi-view stereo (MVS), which estimates dense depth. The reference open-source system is COLMAP: "Structure-from-Motion Revisited" (Schönberger and Frahm, CVPR 2016) and "Pixelwise View Selection for Unstructured Multi-View Stereo" (Schönberger, Zheng, Frahm and Pollefeys, ECCV 2016). Deep learning replaces individual stages, for example MVSNet (Yao, Luo, Li, Fang and Quan, ECCV 2018) for dense depth.
Input: a set of photos of a static scene, often with unknown poses and sometimes unknown intrinsics. Output: camera parameters plus a point cloud, mesh, implicit surface or radiance field. The difficulty is that projection discards depth. Depth comes back only through parallax: the same point seen from two or more positions.
Each camera is a pinhole. A world point projects to pixel through the intrinsics and the pose . SfM inverts this for all cameras and points at once, by minimising the reprojection error in bundle adjustment (Triggs et al., 2000):
- intrinsics: focal lengths in pixels and principal point
- perspective division ; is the depth of the point
- observations: point was matched in image at pixel
- robust loss (Huber, Cauchy) so that wrong matches do not dominate
Structure from motion, then multi-view stereo
SfM starts with local features: SIFT keypoints with 128-D descriptors (Lowe, 2004), matched by nearest neighbour with Lowe's ratio test. Matches between two views must satisfy the epipolar constraint with in normalised coordinates. RANSAC with the five-point algorithm (Nistér, 2004) estimates and rejects outliers. Incremental SfM then grows the model one image at a time: register the new camera from 2D–3D matches (PnP), triangulate new points, and run bundle adjustment.
MVS assumes the poses are known. Classical MVS such as COLMAP's PatchMatch stereo searches, for every pixel, the depth and normal whose patch looks most alike in the other views (normalised cross-correlation). It propagates good hypotheses to neighbouring pixels. Depth maps are then fused into a point cloud by keeping only depths that are consistent across views.
Depth accuracy follows from triangulation geometry. For two parallel cameras with baseline , depth comes from the disparity , and its error grows with the square of the depth:
- focal length in pixels
- baseline: distance between the camera centres, in metres
- disparity error; with independent noise on each pixel coordinate,
Learned multi-view stereo
MVSNet keeps the geometry and learns the matching. A shared 2D CNN extracts 32-channel features at a quarter of the input resolution. Each source feature map is warped into the reference view by the homography of each of depth planes. The warped volumes are merged by their variance, which works for any number of views. A 3D CNN regularises the cost volume, a softmax along depth gives , and the depth is its expectation:
- features of view warped onto the planes, shape
- low variance at a plane means the views agree, so the true depth is likely there
- soft argmin: differentiable, and sub-plane accurate; trained with an L1 loss to ground-truth depth
The cost volume costs memory proportional to . Later methods (R-MVSNet with recurrent regularisation, cascade cost volumes with coarse-to-fine depth ranges) reduce it. A different line drops the explicit pipeline. DUSt3R (Wang et al., CVPR 2024) regresses a pointmap, a 3D point per pixel in the first camera's frame, directly from an uncalibrated image pair with a transformer. Poses and intrinsics are then read off the pointmaps.
| Representation | What is stored | Strength | Weakness |
|---|---|---|---|
| Point cloud | xyz (+ colour, normal) per point | direct output of MVS and SfM | no surface, holes |
| Voxel grid | occupancy or TSDF per cell | simple 3D CNNs (3D-R2N2, Choy et al. 2016) | cubic memory: 256³ cells = 16.8M = 67 MB |
| Mesh | vertices + triangles | compact, renders everywhere (Pixel2Mesh deforms an ellipsoid, Wang et al. 2018) | fixed topology is hard to learn |
| Implicit surface | network f(x) = signed distance or occupancy | continuous resolution (DeepSDF, Occupancy Networks, both CVPR 2019) | needs marching cubes to extract a mesh |
| Radiance field | density + view-dependent colour (MLP or Gaussians) | photorealistic novel views | geometry is a by-product, can be noisy |
Radiance fields are covered in Neural radiance fields (NeRF) and 3D Gaussian splatting. Both are optimised per scene, and both normally take their camera poses (and, for 3DGS, the initial points) from COLMAP. Monocular depth networks are covered in Depth estimation.
Benchmarks and metrics
- DTU
- indoor objects captured by a robot-arm camera with structured-light ground truth (Aanæs et al., 2016). Reports accuracy (prediction → ground truth distance), completeness (ground truth → prediction) and their mean, in mm.
- Tanks and Temples
- real indoor/outdoor scenes with laser-scanned ground truth (Knapitsch, Park, Zhou and Koltun, SIGGRAPH 2017). Reports the F-score of precision and recall at a distance threshold τ.
- Learned matching
- SuperPoint keypoints (DeTone et al., 2018) with the SuperGlue graph-network matcher (Sarlin et al., CVPR 2020) are a common drop-in for SIFT + ratio test inside SfM.
Worked example: stereo triangulation
| Quantity | Formula | Value |
|---|---|---|
| u₁ (left camera) | f·(X − c₁)/Z = 500·0.65/5 | 65 px |
| u₂ (right camera) | f·(X − c₂)/Z = 500·0.15/5 | 15 px |
| disparity d | u₁ − u₂ | 50 px |
| depth Z | f·b/d = 250/50 | 5.000 m |
| lateral X | c₁ + Z·u₁/f = −0.25 + 0.65 | 0.400 m |
| Z if d is off by 1 px | 250/49 | 5.102 m (error 10.2 cm) |
| same at Z = 10 m (d = 25 px) | 250/24 | 10.417 m (error 41.7 cm) |
The linear rule gives 10 cm and 40 cm for these two cases. A one-pixel error costs four times more depth at twice the distance. This is why outdoor scenes need wide baselines, and why distant points in SfM have large uncertainty along the viewing ray.
1from pathlib import Path2import pycolmap34images, out = Path("images"), Path("out")5out.mkdir(exist_ok=True)6db = out / "database.db"78pycolmap.extract_features(db, images) # SIFT keypoints + descriptors9pycolmap.match_exhaustive(db) # all pairs, ratio test, RANSAC on E/F10maps = pycolmap.incremental_mapping(db, images, out) # PnP + triangulation + bundle adjustment11rec = maps[0]12rec.write(out)13print(rec.summary()) # registered images, points, mean reprojection error1415mvs = out / "mvs"16pycolmap.undistort_images(mvs, out, images) # (output, sparse model, images)17pycolmap.patch_match_stereo(mvs) # per-view depth and normal maps18pycolmap.stereo_fusion(mvs / "dense.ply", mvs) # fuse consistent depths into a dense cloudCheck yourself
A stereo rig measures a point at 4 m with a depth error of 2 cm. With the same cameras and matching noise, what error do you expect at 12 m?
Check yourself
Why does MVSNet merge the warped feature volumes with a variance instead of concatenating them?
Remember
- ✓SfM: features → matching with RANSAC on the essential matrix → incremental PnP registration, triangulation and bundle adjustment of the reprojection error. COLMAP is the standard implementation.
- ✓MVS: with poses fixed, estimate per-pixel depth (PatchMatch photo-consistency, or a learned cost volume) and fuse the consistent depth maps.
- ✓Stereo depth and its error : error grows with the square of distance and falls with baseline.
- ✓MVSNet: homography warping onto D depth planes, variance cost volume, 3D CNN, soft argmin depth.
- ✓Representations trade memory and topology: points, voxels (cubic memory), meshes, implicit SDF/occupancy, radiance fields (NeRF, 3DGS on COLMAP poses).