Depth estimation

Depth estimation predicts per-pixel distance. Stereo triangulates Z = f·B/d; monocular networks learn it from LiDAR with a scale-invariant log loss, or self-supervised from video via photometric reprojection. Scored by AbsRel and δ < 1.25.

8 min read774 wordscore0 viewsupdated 2026-10-15

Definition

Depth estimation predicts, for every pixel, the distance ZZ along the camera axis to the visible surface. Stereo methods triangulate it from two calibrated views. Monocular methods infer it from one image, first supervised (Eigen, Puhrsch and Fergus, "Depth Map Prediction from a Single Image using a Multi-Scale Deep Network", NeurIPS 2014) and then self-supervised from stereo pairs or video (Garg et al., ECCV 2016; Godard, Mac Aodha and Brostow, CVPR 2017; Zhou et al., CVPR 2017).

The output is a dense H×WH\times W map in metres, or a relative map that is correct only up to scale (and sometimes shift). Robots, driver assistance, AR occlusion, portrait blur and 3D photo effects all use it. A single image is geometrically ambiguous: a toy car up close and a real car far away project to the same pixels. A monocular network can only learn depth from cues such as known object sizes, perspective, texture gradients, occlusion and the height of the ground contact point.

Stereo: depth from disparity

After rectification, a point appears on the same row in both images, shifted by the disparity d=xL−xRd=x_L-x_R. Similar triangles give depth directly:

Z=f Bd,ΔZ≈Z2f B ΔdZ=\frac{f\,B}{d},\qquad \Delta Z\approx\frac{Z^2}{f\,B}\,\Delta d
ff
focal length in pixels (about 721.5 px for KITTI's rectified cameras)
BB
baseline between the camera centres (0.54 m on KITTI)
Δd\Delta d
disparity error; the depth error grows with the square of the distance

The hard part is matching: finding which right-image pixel corresponds to each left-image pixel, even in textureless, reflective or occluded areas. Deep stereo networks build a cost volume of matching costs over candidate disparities and regress dd from it. Monocular networks skip matching entirely.

stereo (top view)P (depth Z)xLxRfbaseline Bd = xL − xRZ = f·B / df = 721.5 pxB = 0.54 md = 10 px→ Z = 38.96 m1 px error → +4.33 msupervised monocular (Eigen et al. 2014 → DPT)RGB, 1 frameenc–decdepth d (H×W)loss vsLiDAR d*scale-invariant log loss, dᵢ = log yᵢ − log yᵢ*:(1/n)Σdᵢ² − (λ/n²)(Σdᵢ)², λ = 0.5 in trainingLiDAR is sparse: loss only where d* existsMiDaS/DPT: mixed datasets → relative depth,scale- and shift-invariant loss on disparityself-supervised from video (SfMLearner 2017, Monodepth2 2019): no depth labelstarget Iₜsource Iₛ(t ± 1)DepthNet→ DₜPoseNet(Iₜ,Iₛ)→ Tₜ→ₛpₛ ∼ K Tₜ→ₛ Dₜ(pₜ) K⁻¹pₜproject every pixelsample Iₛat pₛ → Îₜphotometricpe(Iₜ, Îₜ)+ smoothnesspe = (α/2)(1 − SSIM(Iₜ, Îₜ)) + (1 − α)‖Iₜ − Îₜ‖₁, α = 0.85 (Monodepth)Monodepth2: per-pixel minimum over source frames (occlusion), auto-mask static pixelsstereo training: T is the known baseline, so depth comes out in metres; video-only is up to scale
fig.Three routes to depth. Stereo: with f = 721.5 px and B = 0.54 m, a disparity of 10 px means 38.96 m, and a 1 px matching error moves it to 43.29 m. Supervised monocular: an encoder–decoder maps one RGB frame to a depth map, trained only where the sparse LiDAR has points, with a scale-invariant log loss. Self-supervised: DepthNet and PoseNet let every target pixel be projected into a neighbouring frame, and a photometric loss compares the warped source with the target, so no depth labels are needed.

Supervised monocular: the scale-invariant loss

Eigen et al. stacked a coarse network, which sees the whole image through fully connected layers, and a fine network that refines local detail. Because absolute scale is the most ambiguous part of the prediction, they measured error in log space and discounted a global offset:

D(y,y∗)=1n∑idi2−λn2(∑idi)2,di=log⁡yi−log⁡yi∗D(y,y^*)=\frac1n\sum_i d_i^2-\frac{\lambda}{n^2}\Big(\sum_i d_i\Big)^2,\qquad d_i=\log y_i-\log y_i^*
y, y∗y,\ y^*
predicted and ground-truth depth at the n pixels that have ground truth
λ=1\lambda=1
the variance of dd: multiplying every prediction by ss adds log⁡s\log s to every did_i and leaves DD unchanged
λ=0.5\lambda=0.5
used for training: half credit for getting the overall scale right, which keeps predictions in metres

Later supervised models replaced the backbone and the loss. Laina et al. (3DV 2016) used a ResNet-50 encoder with up-projection blocks and a reverse Huber loss. DPT (Ranftl, Bochkovskiy and Koltun, ICCV 2021) used a Vision Transformers (ViT) encoder. MiDaS (Ranftl et al., TPAMI 2022) trained on a mix of datasets with incompatible scales. It predicts relative inverse depth with a loss that is invariant to scale and shift, which makes it generalise zero-shot across scenes. Depth Anything (Yang et al., CVPR 2024) scaled this up with about 1.5M labelled and 62M unlabelled images, using pseudo-labels from a teacher.

Self-supervised: view synthesis as the signal

LiDAR labels are expensive and sparse. If the depth DtD_t and the camera motion Tt→sT_{t\to s} are right, the pixels of a nearby frame IsI_s can be warped to reproduce the target frame ItI_t. The warp is differentiable thanks to bilinear sampling, so a photometric error trains both networks:

ps∼K Tt→s Dt(pt) K−1pt,pe(It,I^t)=α2(1−SSIM(It,I^t))+(1−α)∥It−I^t∥1p_s\sim K\,T_{t\to s}\,D_t(p_t)\,K^{-1}p_t,\qquad pe(I_t,\hat I_t)=\frac{\alpha}{2}\big(1-\mathrm{SSIM}(I_t,\hat I_t)\big)+(1-\alpha)\big\|I_t-\hat I_t\big\|_1
KK
camera intrinsics; ptp_t is a pixel in homogeneous coordinates
I^t\hat I_t
IsI_s sampled at the projected locations psp_s
α=0.85\alpha=0.85
SSIM/L1 balance from Monodepth; an edge-aware smoothness term on disparity is added

With a stereo pair, TT is the known baseline, so depth comes out in metres. With video, PoseNet learns TT, and depth is recovered only up to an unknown scale. Monodepth2 (Godard et al., ICCV 2019) fixed the main failure cases. It takes the per-pixel minimum of the error over source frames, which handles occlusion, and auto-masks pixels that do not change between frames, which handles a static camera or objects moving with it. Moving objects still break the static-scene assumption.

Benchmarks and evaluation protocol

KITTI (Eigen split)
driving scenes, sparse Velodyne LiDAR ground truth; 697 test images, depth capped at 80 m
NYU Depth v2
indoor Kinect RGB-D; 1,449 densely labelled frames, 654 used for testing; depth up to about 10 m
Metrics
AbsRel =1n∑∣d−d∗∣/d∗=\frac1n\sum|d-d^*|/d^*, SqRel, RMSE, RMSE log, and the accuracy δ<1.25k\delta<1.25^k: the share of pixels with max⁡(d/d∗,d∗/d)<1.25k\max(d/d^*,d^*/d)<1.25^k for k=1,2,3k=1,2,3
Median scaling
monocular video models are evaluated after multiplying each prediction by median(d*)/median(d), because their scale is unknown
Monodepth2 (video only, 640×192)
AbsRel 0.115, δ < 1.25 of 0.877 on the KITTI Eigen split, with median scaling
prediction = 1.00 × true depth × exp(0.10·n)ground truth d* (LiDAR-like, sky has none)monocular prediction dcolour: log depth, 4 m strong → 80 m faint · grey: no ground truth, not scored · 457 pixels scoredAbsRel mean |d−d*|/d*0.084↓RMSE (m)2.88↓δ < 1.2596.9%↑SI log, λ = 0 (log MSE)0.0109↓SI log, λ = 0.5 (Eigen loss)0.0109↓SI log, λ = 1 (scale-free)0.0109↓λ = 1 and median scaling both ignore global scalelog d vs log d*d* →band: δ < 1.25
median scaling
fig.A road scene with ground truth on 457 pixels (4.4–80 m, sky unscored). The prediction is the true depth times a global scale times exp(σ·n). At the defaults (scale 1, σ = 0.10): AbsRel 0.084, RMSE 2.88 m, δ < 1.25 96.9%, and all three scale-invariant errors equal 0.0109. At scale 1.5, AbsRel jumps to 0.515 and δ falls to 3.1%. The λ = 0.5 loss rises to 0.0948 while λ = 1 stays 0.0109. Median scaling (×0.682) brings AbsRel back to 0.088 and δ to 95.2%, the same result as for any other scale. Raising σ to 0.20 gives AbsRel 0.170 and δ 72.9%.

Worked example: four pixels

Ground truth d∗=(2,4,10,40)d^*=(2,4,10,40) m. Prediction A is everything 20% too far. Prediction B is exact except one far pixel at 60 m instead of 40 m.

PredictionAbsRelRMSE (m)δ < 1.25SI, λ = 0 / 0.5 / 1
A: (2.4, 4.8, 12, 48)0.2004.15100%0.0332 / 0.0166 / 0
A after median scaling (×0.833)00100%0 / 0 / 0
B: (2, 4, 10, 60)0.12510.0075%0.0411 / 0.0360 / 0.0308
Metrics computed in Python (SI = scale-invariant log error)

A passes δ<1.25\delta<1.25 on every pixel while being 20% wrong everywhere, because the threshold allows 25%. B has lower AbsRel but much higher RMSE, because RMSE is dominated by far pixels. That is why papers report several metrics together.

python
1import numpy as np
2from transformers import pipeline
3
4def depth_metrics(pred, gt, cap=80.0):
5 v = (gt > 1e-3) & (gt < cap) # only pixels with LiDAR returns
6 p, g = np.clip(pred[v], 1e-3, cap), gt[v]
7 ratio = np.maximum(p / g, g / p)
8 return {"abs_rel": np.mean(np.abs(p - g) / g),
9 "rmse": np.sqrt(np.mean((p - g) ** 2)),
10 "delta1": np.mean(ratio < 1.25)}
11
12def align_disparity(disp, gt):
13 """DPT/MiDaS output inverse depth up to scale and shift: fit s, t on 1/gt."""
14 v = gt > 0
15 A = np.stack([disp[v], np.ones(v.sum())], 1)
16 s, t = np.linalg.lstsq(A, 1.0 / gt[v], rcond=None)[0]
17 return 1.0 / np.clip(s * disp + t, 1e-6, None)
18
19estimator = pipeline("depth-estimation", model="Intel/dpt-large")
20out = estimator(image) # PIL image
21disp = out["predicted_depth"].squeeze().numpy() # relative inverse depth, model resolution
22# resize disp to the LiDAR map's H×W before aligning
23print(depth_metrics(align_disparity(disp, lidar_depth), lidar_depth))
Relative depth with DPT via Hugging Face, aligned to LiDAR and scored (the metric and alignment functions were run on the worked example; the model call was not)

Where it's used

Driving and robotics

free-space and obstacle distance from stereo rigs or cheap monocular cameras; LiDAR supervision or self-supervision from fleet video

Phones and AR

portrait-mode blur, occlusion of virtual objects, and 3D photo effects from one image

3D and generation

initialisation for 3D reconstruction and depth conditioning for diffusion editors (ControlNet depth maps)

Check yourself

A stereo rig has f = 700 px and B = 0.5 m. A point has disparity 7 px. How far away is it, and roughly how much does a 1 px disparity error change that?

Check yourself

Every predicted depth is multiplied by 2. Which value stays the same?

Remember

  • ✓Stereo: Z=fB/dZ=fB/d and ΔZ≈Z2Δd/(fB)\Delta Z\approx Z^2\Delta d/(fB), so accuracy falls with the square of distance.
  • ✓Monocular depth is ambiguous in scale. Eigen et al. (2014) trained with 1n∑di2−λn2(∑di)2\frac1n\sum d_i^2-\frac{\lambda}{n^2}(\sum d_i)^2 in log space, λ=0.5\lambda=0.5.
  • ✓Self-supervised: DepthNet and PoseNet are trained by warping a neighbouring frame into the target view under an SSIM + L1 photometric loss (α = 0.85).
  • ✓Video-only models are evaluated with median scaling. MiDaS, DPT and Depth Anything predict relative depth that needs scale and shift alignment.
  • ✓Report AbsRel, RMSE and δ < 1.25 together: δ allows 25% error, and RMSE is dominated by far pixels.
← back to Deep Learning roadmap