Definition
Depth estimation predicts, for every pixel, the distance along the camera axis to the visible surface. Stereo methods triangulate it from two calibrated views. Monocular methods infer it from one image, first supervised (Eigen, Puhrsch and Fergus, "Depth Map Prediction from a Single Image using a Multi-Scale Deep Network", NeurIPS 2014) and then self-supervised from stereo pairs or video (Garg et al., ECCV 2016; Godard, Mac Aodha and Brostow, CVPR 2017; Zhou et al., CVPR 2017).
The output is a dense map in metres, or a relative map that is correct only up to scale (and sometimes shift). Robots, driver assistance, AR occlusion, portrait blur and 3D photo effects all use it. A single image is geometrically ambiguous: a toy car up close and a real car far away project to the same pixels. A monocular network can only learn depth from cues such as known object sizes, perspective, texture gradients, occlusion and the height of the ground contact point.
Stereo: depth from disparity
After rectification, a point appears on the same row in both images, shifted by the disparity . Similar triangles give depth directly:
- focal length in pixels (about 721.5 px for KITTI's rectified cameras)
- baseline between the camera centres (0.54 m on KITTI)
- disparity error; the depth error grows with the square of the distance
The hard part is matching: finding which right-image pixel corresponds to each left-image pixel, even in textureless, reflective or occluded areas. Deep stereo networks build a cost volume of matching costs over candidate disparities and regress from it. Monocular networks skip matching entirely.
Supervised monocular: the scale-invariant loss
Eigen et al. stacked a coarse network, which sees the whole image through fully connected layers, and a fine network that refines local detail. Because absolute scale is the most ambiguous part of the prediction, they measured error in log space and discounted a global offset:
- predicted and ground-truth depth at the n pixels that have ground truth
- the variance of : multiplying every prediction by adds to every and leaves unchanged
- used for training: half credit for getting the overall scale right, which keeps predictions in metres
Later supervised models replaced the backbone and the loss. Laina et al. (3DV 2016) used a ResNet-50 encoder with up-projection blocks and a reverse Huber loss. DPT (Ranftl, Bochkovskiy and Koltun, ICCV 2021) used a Vision Transformers (ViT) encoder. MiDaS (Ranftl et al., TPAMI 2022) trained on a mix of datasets with incompatible scales. It predicts relative inverse depth with a loss that is invariant to scale and shift, which makes it generalise zero-shot across scenes. Depth Anything (Yang et al., CVPR 2024) scaled this up with about 1.5M labelled and 62M unlabelled images, using pseudo-labels from a teacher.
Self-supervised: view synthesis as the signal
LiDAR labels are expensive and sparse. If the depth and the camera motion are right, the pixels of a nearby frame can be warped to reproduce the target frame . The warp is differentiable thanks to bilinear sampling, so a photometric error trains both networks:
- camera intrinsics; is a pixel in homogeneous coordinates
- sampled at the projected locations
- SSIM/L1 balance from Monodepth; an edge-aware smoothness term on disparity is added
With a stereo pair, is the known baseline, so depth comes out in metres. With video, PoseNet learns , and depth is recovered only up to an unknown scale. Monodepth2 (Godard et al., ICCV 2019) fixed the main failure cases. It takes the per-pixel minimum of the error over source frames, which handles occlusion, and auto-masks pixels that do not change between frames, which handles a static camera or objects moving with it. Moving objects still break the static-scene assumption.
Benchmarks and evaluation protocol
- KITTI (Eigen split)
- driving scenes, sparse Velodyne LiDAR ground truth; 697 test images, depth capped at 80 m
- NYU Depth v2
- indoor Kinect RGB-D; 1,449 densely labelled frames, 654 used for testing; depth up to about 10 m
- Metrics
- AbsRel , SqRel, RMSE, RMSE log, and the accuracy : the share of pixels with for
- Median scaling
- monocular video models are evaluated after multiplying each prediction by median(d*)/median(d), because their scale is unknown
- Monodepth2 (video only, 640×192)
- AbsRel 0.115, δ < 1.25 of 0.877 on the KITTI Eigen split, with median scaling
Worked example: four pixels
Ground truth m. Prediction A is everything 20% too far. Prediction B is exact except one far pixel at 60 m instead of 40 m.
| Prediction | AbsRel | RMSE (m) | δ < 1.25 | SI, λ = 0 / 0.5 / 1 |
|---|---|---|---|---|
| A: (2.4, 4.8, 12, 48) | 0.200 | 4.15 | 100% | 0.0332 / 0.0166 / 0 |
| A after median scaling (×0.833) | 0 | 0 | 100% | 0 / 0 / 0 |
| B: (2, 4, 10, 60) | 0.125 | 10.00 | 75% | 0.0411 / 0.0360 / 0.0308 |
A passes on every pixel while being 20% wrong everywhere, because the threshold allows 25%. B has lower AbsRel but much higher RMSE, because RMSE is dominated by far pixels. That is why papers report several metrics together.
1import numpy as np2from transformers import pipeline34def depth_metrics(pred, gt, cap=80.0):5 v = (gt > 1e-3) & (gt < cap) # only pixels with LiDAR returns6 p, g = np.clip(pred[v], 1e-3, cap), gt[v]7 ratio = np.maximum(p / g, g / p)8 return {"abs_rel": np.mean(np.abs(p - g) / g),9 "rmse": np.sqrt(np.mean((p - g) ** 2)),10 "delta1": np.mean(ratio < 1.25)}1112def align_disparity(disp, gt):13 """DPT/MiDaS output inverse depth up to scale and shift: fit s, t on 1/gt."""14 v = gt > 015 A = np.stack([disp[v], np.ones(v.sum())], 1)16 s, t = np.linalg.lstsq(A, 1.0 / gt[v], rcond=None)[0]17 return 1.0 / np.clip(s * disp + t, 1e-6, None)1819estimator = pipeline("depth-estimation", model="Intel/dpt-large")20out = estimator(image) # PIL image21disp = out["predicted_depth"].squeeze().numpy() # relative inverse depth, model resolution22# resize disp to the LiDAR map's H×W before aligning23print(depth_metrics(align_disparity(disp, lidar_depth), lidar_depth))Where it's used
Driving and robotics
free-space and obstacle distance from stereo rigs or cheap monocular cameras; LiDAR supervision or self-supervision from fleet video
Phones and AR
portrait-mode blur, occlusion of virtual objects, and 3D photo effects from one image
3D and generation
initialisation for 3D reconstruction and depth conditioning for diffusion editors (ControlNet depth maps)
Check yourself
A stereo rig has f = 700 px and B = 0.5 m. A point has disparity 7 px. How far away is it, and roughly how much does a 1 px disparity error change that?
Check yourself
Every predicted depth is multiplied by 2. Which value stays the same?
Remember
- ✓Stereo: and , so accuracy falls with the square of distance.
- ✓Monocular depth is ambiguous in scale. Eigen et al. (2014) trained with in log space, .
- ✓Self-supervised: DepthNet and PoseNet are trained by warping a neighbouring frame into the target view under an SSIM + L1 photometric loss (α = 0.85).
- ✓Video-only models are evaluated with median scaling. MiDaS, DPT and Depth Anything predict relative depth that needs scale and shift alignment.
- ✓Report AbsRel, RMSE and δ < 1.25 together: δ allows 25% error, and RMSE is dominated by far pixels.