# Depth estimation

> Depth estimation predicts per-pixel distance. Stereo triangulates Z = f·B/d; monocular networks learn it from LiDAR with a scale-invariant log loss, or self-supervised from video via photometric reprojection. Scored by AbsRel and δ < 1.25.

Source: https://www.mlfoundry.in/learn/depth-estimation  
Section: Deep Learning › Deep Learning for Computer Vision  
Level: core · 8 min read · updated 2026-10-15

**Depth estimation:** predicts, for every pixel, the distance $Z$ along the camera axis to the visible surface. **Stereo** methods triangulate it from two calibrated views. **Monocular** methods infer it from one image, first supervised (Eigen, Puhrsch and Fergus, "Depth Map Prediction from a Single Image using a Multi-Scale Deep Network", NeurIPS 2014) and then **self-supervised** from stereo pairs or video (Garg et al., ECCV 2016; Godard, Mac Aodha and Brostow, CVPR 2017; Zhou et al., CVPR 2017).

The output is a dense $H\times W$ map in metres, or a **relative** map that is correct only up to scale (and sometimes shift). Robots, driver assistance, AR occlusion, portrait blur and 3D photo effects all use it. A single image is geometrically ambiguous: a toy car up close and a real car far away project to the same pixels. A monocular network can only learn depth from cues such as known object sizes, perspective, texture gradients, occlusion and the height of the ground contact point.

## Stereo: depth from disparity

After rectification, a point appears on the same row in both images, shifted by the **disparity** $d=x_L-x_R$. Similar triangles give depth directly:

$$
Z=\frac{f\,B}{d},\qquad \Delta Z\approx\frac{Z^2}{f\,B}\,\Delta d
$$

- $f$: focal length in pixels (about 721.5 px for KITTI's rectified cameras)
- $B$: baseline between the camera centres (0.54 m on KITTI)
- $\Delta d$: disparity error; the depth error grows with the square of the distance

The hard part is matching: finding which right-image pixel corresponds to each left-image pixel, even in textureless, reflective or occluded areas. Deep stereo networks build a **cost volume** of matching costs over candidate disparities and regress $d$ from it. Monocular networks skip matching entirely.

> Figure: Three routes to depth. Stereo: with f = 721.5 px and B = 0.54 m, a disparity of 10 px means 38.96 m, and a 1 px matching error moves it to 43.29 m. Supervised monocular: an encoder–decoder maps one RGB frame to a depth map, trained only where the sparse LiDAR has points, with a scale-invariant log loss. Self-supervised: DepthNet and PoseNet let every target pixel be projected into a neighbouring frame, and a photometric loss compares the warped source with the target, so no depth labels are needed.

## Supervised monocular: the scale-invariant loss

Eigen et al. stacked a coarse network, which sees the whole image through fully connected layers, and a fine network that refines local detail. Because absolute scale is the most ambiguous part of the prediction, they measured error in log space and discounted a global offset:

$$
D(y,y^*)=\frac1n\sum_i d_i^2-\frac{\lambda}{n^2}\Big(\sum_i d_i\Big)^2,\qquad d_i=\log y_i-\log y_i^*
$$

- $y,\ y^*$: predicted and ground-truth depth at the n pixels that have ground truth
- $\lambda=1$: the variance of $d$: multiplying every prediction by $s$ adds $\log s$ to every $d_i$ and leaves $D$ unchanged
- $\lambda=0.5$: used for training: half credit for getting the overall scale right, which keeps predictions in metres

Later supervised models replaced the backbone and the loss. Laina et al. (3DV 2016) used a ResNet-50 encoder with up-projection blocks and a reverse Huber loss. DPT (Ranftl, Bochkovskiy and Koltun, ICCV 2021) used a **Vision Transformers (ViT)** encoder. MiDaS (Ranftl et al., TPAMI 2022) trained on a mix of datasets with incompatible scales. It predicts relative inverse depth with a loss that is invariant to scale and shift, which makes it generalise zero-shot across scenes. Depth Anything (Yang et al., CVPR 2024) scaled this up with about 1.5M labelled and 62M unlabelled images, using pseudo-labels from a teacher.

## Self-supervised: view synthesis as the signal

LiDAR labels are expensive and sparse. If the depth $D_t$ and the camera motion $T_{t\to s}$ are right, the pixels of a nearby frame $I_s$ can be warped to reproduce the target frame $I_t$. The warp is differentiable thanks to bilinear sampling, so a photometric error trains both networks:

$$
p_s\sim K\,T_{t\to s}\,D_t(p_t)\,K^{-1}p_t,\qquad pe(I_t,\hat I_t)=\frac{\alpha}{2}\big(1-\mathrm{SSIM}(I_t,\hat I_t)\big)+(1-\alpha)\big\|I_t-\hat I_t\big\|_1
$$

- $K$: camera intrinsics; $p_t$ is a pixel in homogeneous coordinates
- $\hat I_t$: $I_s$ sampled at the projected locations $p_s$
- $\alpha=0.85$: SSIM/L1 balance from Monodepth; an edge-aware smoothness term on disparity is added

With a stereo pair, $T$ is the known baseline, so depth comes out in metres. With video, PoseNet learns $T$, and depth is recovered only up to an unknown scale. Monodepth2 (Godard et al., ICCV 2019) fixed the main failure cases. It takes the **per-pixel minimum** of the error over source frames, which handles occlusion, and **auto-masks** pixels that do not change between frames, which handles a static camera or objects moving with it. Moving objects still break the static-scene assumption.

**Benchmarks and evaluation protocol**

- **KITTI (Eigen split):** driving scenes, sparse Velodyne LiDAR ground truth; 697 test images, depth capped at 80 m
- **NYU Depth v2:** indoor Kinect RGB-D; 1,449 densely labelled frames, 654 used for testing; depth up to about 10 m
- **Metrics:** AbsRel $=\frac1n\sum|d-d^*|/d^*$, SqRel, RMSE, RMSE log, and the accuracy $\delta<1.25^k$: the share of pixels with $\max(d/d^*,d^*/d)<1.25^k$ for $k=1,2,3$
- **Median scaling:** monocular video models are evaluated after multiplying each prediction by median(d*)/median(d), because their scale is unknown
- **Monodepth2 (video only, 640×192):** AbsRel 0.115, δ < 1.25 of 0.877 on the KITTI Eigen split, with median scaling

> Figure: A road scene with ground truth on 457 pixels (4.4–80 m, sky unscored). The prediction is the true depth times a global scale times exp(σ·n). At the defaults (scale 1, σ = 0.10): AbsRel 0.084, RMSE 2.88 m, δ < 1.25 96.9%, and all three scale-invariant errors equal 0.0109. At scale 1.5, AbsRel jumps to 0.515 and δ falls to 3.1%. The λ = 0.5 loss rises to 0.0948 while λ = 1 stays 0.0109. Median scaling (×0.682) brings AbsRel back to 0.088 and δ to 95.2%, the same result as for any other scale. Raising σ to 0.20 gives AbsRel 0.170 and δ 72.9%.

## Worked example: four pixels

Ground truth $d^*=(2,4,10,40)$ m. Prediction A is everything 20% too far. Prediction B is exact except one far pixel at 60 m instead of 40 m.

Metrics computed in Python (SI = scale-invariant log error)

| Prediction | AbsRel | RMSE (m) | δ < 1.25 | SI, λ = 0 / 0.5 / 1 |
| --- | --- | --- | --- | --- |
| A: (2.4, 4.8, 12, 48) | 0.200 | 4.15 | 100% | 0.0332 / 0.0166 / 0 |
| A after median scaling (×0.833) | 0 | 0 | 100% | 0 / 0 / 0 |
| B: (2, 4, 10, 60) | 0.125 | 10.00 | 75% | 0.0411 / 0.0360 / 0.0308 |

A passes $\delta<1.25$ on every pixel while being 20% wrong everywhere, because the threshold allows 25%. B has lower AbsRel but much higher RMSE, because RMSE is dominated by far pixels. That is why papers report several metrics together.

```python
import numpy as np
from transformers import pipeline

def depth_metrics(pred, gt, cap=80.0):
    v = (gt > 1e-3) & (gt < cap)                      # only pixels with LiDAR returns
    p, g = np.clip(pred[v], 1e-3, cap), gt[v]
    ratio = np.maximum(p / g, g / p)
    return {"abs_rel": np.mean(np.abs(p - g) / g),
            "rmse": np.sqrt(np.mean((p - g) ** 2)),
            "delta1": np.mean(ratio < 1.25)}

def align_disparity(disp, gt):
    """DPT/MiDaS output inverse depth up to scale and shift: fit s, t on 1/gt."""
    v = gt > 0
    A = np.stack([disp[v], np.ones(v.sum())], 1)
    s, t = np.linalg.lstsq(A, 1.0 / gt[v], rcond=None)[0]
    return 1.0 / np.clip(s * disp + t, 1e-6, None)

estimator = pipeline("depth-estimation", model="Intel/dpt-large")
out = estimator(image)                                   # PIL image
disp = out["predicted_depth"].squeeze().numpy()          # relative inverse depth, model resolution
# resize disp to the LiDAR map's H×W before aligning
print(depth_metrics(align_disparity(disp, lidar_depth), lidar_depth))
```

Relative depth with DPT via Hugging Face, aligned to LiDAR and scored (the metric and alignment functions were run on the worked example; the model call was not)

> **Relative depth is not metric depth:** MiDaS, DPT and similar models output relative inverse depth. Treating it as metres, or comparing it with metric models without stating the alignment (median scale, or least-squares scale and shift), gives meaningless numbers. For the same reason, check whether a KITTI result used median scaling and the 80 m cap before comparing it with another.

> **Why stereo degrades with distance:** Depth error grows as $Z^2/(fB)$. On KITTI geometry ($fB\approx389.6$ px·m), a 1 px matching error costs 0.25 m at 9.74 m but 4.33 m at 38.96 m. Wider baselines and longer focal lengths push the usable range out, and LiDAR or radar complement cameras at long range.

- **Driving and robotics:** free-space and obstacle distance from stereo rigs or cheap monocular cameras; LiDAR supervision or self-supervision from fleet video
- **Phones and AR:** portrait-mode blur, occlusion of virtual objects, and 3D photo effects from one image
- **3D and generation:** initialisation for **3D reconstruction** and depth conditioning for diffusion editors (ControlNet depth maps)

**Question:** A stereo rig has f = 700 px and B = 0.5 m. A point has disparity 7 px. How far away is it, and roughly how much does a 1 px disparity error change that?

- A. 50 m, about 8 m
- B. 5 m, about 0.07 m
- C. 50 m, about 0.7 m
- D. 350 m, about 1 m

**Answer:** A. 50 m, about 8 m — $Z=fB/d=350/7=50$ m. $\Delta Z\approx Z^2/(fB)\cdot\Delta d=2500/350\approx7.1$ m. Exactly, $350/6-50=8.3$ m. Far points have small disparities, so each pixel of error is costly.

**Question:** Every predicted depth is multiplied by 2. Which value stays the same?

- A. AbsRel
- B. RMSE
- C. δ < 1.25
- D. The scale-invariant log error with λ = 1

**Answer:** D. The scale-invariant log error with λ = 1 — All $d_i=\log y_i-\log y_i^*$ shift by $\log 2$. With $\lambda=1$, $D$ is the variance of $d$, which a constant shift does not change. The other metrics compare absolute values and get much worse.

**Key points**

- Stereo: $Z=fB/d$ and $\Delta Z\approx Z^2\Delta d/(fB)$, so accuracy falls with the square of distance.
- Monocular depth is ambiguous in scale. Eigen et al. (2014) trained with $\frac1n\sum d_i^2-\frac{\lambda}{n^2}(\sum d_i)^2$ in log space, $\lambda=0.5$.
- Self-supervised: DepthNet and PoseNet are trained by warping a neighbouring frame into the target view under an SSIM + L1 photometric loss (α = 0.85).
- Video-only models are evaluated with median scaling. MiDaS, DPT and Depth Anything predict relative depth that needs scale and shift alignment.
- Report AbsRel, RMSE and δ < 1.25 together: δ allows 25% error, and RMSE is dominated by far pixels.

---

From MLFoundry (https://www.mlfoundry.in). The web page adds interactive figures and diagrams.
