Text-to-3D generation

Text-to-3D either distils a frozen 2-D diffusion model into a NeRF or Gaussian scene with score distillation sampling (DreamFusion, 2022), or generates consistent multi-view images and reconstructs them feed-forward in seconds.

8 min read631 wordsadvanced0 viewsupdated 2026-10-21

Definition

Text-to-3D generation produces a 3-D asset (a radiance field, a set of 3-D Gaussians or a textured mesh) from a text prompt. The founding method is DreamFusion (Poole, Jain, Barron and Mildenhall, Google, 2022), which introduced score distillation sampling (SDS): optimise a 3-D scene so that its renders from random cameras look plausible to a frozen text-to-image diffusion model.

The obstacle is data. Text–image pairs number in the billions. Captioned 3-D assets number in the hundreds of thousands to millions: Objaverse (Deitke et al., 2023) has over 800K objects and Objaverse-XL over 10M, of very mixed quality. Point-E (Nichol et al., OpenAI, 2022) trained directly on 3-D data and produced point clouds in 1–2 minutes on one GPU, but its quality and prompt range lagged image models.

DreamFusion's answer was to need no 3-D data at all. A 2-D model already knows what a corgi looks like from the side and from above. If every render of a 3-D scene looks like a corgi from that angle, the scene is a 3-D corgi. The scene representation is one of the DL articles: Neural radiance fields (NeRF) in DreamFusion, 3D Gaussian splatting in faster successors such as DreamGaussian (2023), usually converted to a mesh at the end.

Score distillation sampling

Render an image x=g(θ,c)x=g(\theta,c) of scene parameters θ\theta from camera cc. Noise it as the diffusion model was trained to see, ask the frozen model to predict the noise, and push the render in the direction that makes the prediction match. DreamFusion showed that the gradient of the diffusion loss with the U-Net Jacobian dropped is a well-behaved update:

∇θLSDS=Et,ε,c[w(t) (ε^ϕ(zt; y, t)−ε) ∂x∂θ],zt=αt x+σt ε\nabla_\theta\mathcal L_{\text{SDS}}=\mathbb E_{t,\varepsilon,c}\Big[w(t)\,\big(\hat\varepsilon_\phi(z_t;\,y,\,t)-\varepsilon\big)\,\frac{\partial x}{\partial\theta}\Big],\qquad z_t=\alpha_t\,x+\sigma_t\,\varepsilon
ε^ϕ\hat\varepsilon_\phi
noise predicted by the frozen diffusion model ϕ\phi for prompt yy; no gradient flows through it
ε\varepsilon
the Gaussian noise that was actually added
w(t)w(t)
timestep weighting; σt2=1−αˉt\sigma_t^2=1-\bar\alpha_t is a common choice
∂x/∂θ\partial x/\partial\theta
backpropagation through the differentiable renderer only
ε^ϕ(zt;y,t)=εϕ(zt;∅,t)+ω (εϕ(zt;y,t)−εϕ(zt;∅,t))\hat\varepsilon_\phi(z_t;y,t)=\varepsilon_\phi(z_t;\varnothing,t)+\omega\,\big(\varepsilon_\phi(z_t;y,t)-\varepsilon_\phi(z_t;\varnothing,t)\big)
ω\omega
classifier-free guidance scale. Image sampling uses about 7.5; DreamFusion used ω=100\omega=100
∅\varnothing
empty prompt (unconditional prediction)

SDS is mode-seeking. It minimises a KL divergence between a Gaussian around the render and the model's noisy distribution, so it climbs towards one high-density image rather than sampling. Large ω\omega sharpens the target so that the gradients from different views agree. That is why DreamFusion needs 100. The side effects are well documented: over-saturated colours, over-smooth surfaces and little diversity across seeds. ProlificDreamer (Wang et al., 2023) replaced SDS with variational score distillation. It swaps the true noise ε\varepsilon for the prediction of a LoRA-tuned copy of the model that tracks the current renders, and works at ordinary guidance (7.5).

score distillation sampling: a 2-D prior supervises a 3-D scene3-D scene θNeRF or 3-DGaussiansrenderx = g(θ, c)64×64 (DreamFusion)add noisez_t = α_t·x + σ_t·εt ~ U(0.02, 0.98)frozen 2-D diffusionε_φ(z_t; y, t)no weights updatedrandom camera cazimuth, elevationy = “a DSLR photo ofa corgi, side view”CFG ω = 100viewSDS gradientw(t)·(ε_φ − ε)·∂x/∂θU-Net Jacobian skippedbackprop through the renderer only; repeat ~10⁴ steps, a new camera each step1 · SDS optimisationDreamFusion (2022),Magic3D, ProlificDreamerper-prompt optimisation:~1.5 h (DreamFusion),40 min (Magic3D)risks: Janus faces,over-saturated colours2 · multi-view diffusionZero-1-to-3, MVDream(2023): one model emitsseveral camera-conditioned views thatattend to each other;used as an SDS prior orfed to reconstruction3 · feed-forwardLRM (2023): image →triplane NeRF in ~5 sInstant3D: text → 4views → LRM, ~20 strained on renderedObjaverse objects;no per-prompt loop
fig.One SDS step: pick a random camera, render (DreamFusion renders 64×64 for the Imagen base model), add noise at a random t in (0.02, 0.98), and let the frozen model predict the noise for a view-dependent prompt ("…, side view"). The weighted residual (ε_φ − ε) is backpropagated through the renderer into θ. Thousands of such steps make up one asset. Below are the three families in use: per-prompt SDS optimisation (minutes to hours), multi-view diffusion models, and feed-forward reconstructors that turn a few views into a 3-D asset in seconds.

Reported numbers (from the papers)

DreamFusion (2022)
frozen Imagen 64×64 base model, mip-NeRF 360-style scene, ω = 100, t ~ U(0.02, 0.98); about 1.5 h per prompt on a 4-chip TPUv4 machine
Magic3D (2023)
coarse neural field, then a textured mesh refined with a latent diffusion model at 512×512; about 40 min, reported as 2× faster than DreamFusion
Janus problem
SDS often grows a front face on every side, because each view is judged alone and front views dominate image data; view-dependent prompts only partly fix it
Zero-1-to-3 (2023)
Stable Diffusion fine-tuned on Objaverse renders to synthesise a new view of an object given the relative camera rotation and translation
MVDream (2023)
generates 4 views jointly, with self-attention across the views and camera embeddings; as an SDS prior it largely removes Janus artefacts
LRM (2023)
a transformer of about 500M parameters, trained on about 1M objects, maps one image to a triplane NeRF in about 5 s
Instant3D (2023)
text → 2×2 grid of 4 views (fine-tuned diffusion) → sparse-view reconstructor; about 20 s per asset
SDS on a toy prior · prompt “a cat” · ω = 1 · step 400 / 400catdogcarunconditional mean16 SDS runs (red)within 0.6 of cat16 / 16mean position(-0.90, 0.44)mean − mode0.120spread (RMS)0.019true samples0.186off chart0prior: 3 modes, s = 0.2“a cat”: weights .90/.05/.05● 16 true cat samples○ start · — path · ● nowθ ← θ − η·σ²(ε_ω − ε)ε_ω = ε_u + ω(ε_c − ε_u)t ~ U[0.02, 0.98], η = 0.05all 16 runs collapsed onto one point near the mode: no diversitytoy: renderer x = θ, exact mixture score; the real prior is a frozen text-to-image U-Nethollow red dots sit on the border: the run has left the chart
guidance ω
fig.SDS on a toy 2-D "image space" with an exact mixture prior (cat, dog, car; prompt weights 0.90/0.05/0.05). Each of 16 random initialisations is one asset. At ω = 1, after 400 steps all 16 runs sit within 0.6 of the cat mode, with mean (−0.90, 0.44), 0.120 from the mode centre. Their spread is 0.019, against 0.186 for 16 true samples: SDS collapses to a mode, it does not sample. At ω = 2 they still collapse (spread 0.018) but the mean moves 0.371 out from the mode, away from the unconditional mean. At ω = 7.5 none are within 0.6 (the mean is 1.837 away), and at ω = 100 every run leaves the chart (9.226 away). This is the toy analogue of over-saturated colours.

Worked example: one SDS step in one dimension

StepFormulaValue
noisy renderz = αx + σε = 0.16 + 0.300.460
noisy prior variancev = α²s² + σ²0.3664
prior score(αμ − z) / v0.4913
predicted noiseε̂ = −σ · score−0.2948
residualε̂ − ε−0.7948
SDS gradientw · (ε̂ − ε)−0.2861
updatex − 0.5 · gradient0.343 (towards 0.8)
Prior for the prompt: N(μ = 0.8, s = 0.1); render x = 0.20; α = 0.8, σ = 0.6, sampled ε = 0.5, w = σ², ω = 1, learning rate 0.5 (computed in Python)

Averaged over ε\varepsilon, the gradient is −w σα(μ−x)/v=−0.2830-w\,\sigma\alpha(\mu-x)/v=-0.2830. It always points at the mode, and the single-sample value −0.2861-0.2861 is close. The noise term ε(σ2/v−1)\varepsilon(\sigma^2/v-1) has coefficient −0.017-0.017 here. Once x=μx=\mu the expected gradient is zero, whatever ss is. That is the collapse the viz shows: the spread of the prior never enters the fixed point.

python
1import torch
2from diffusers import StableDiffusionPipeline
3
4pipe = StableDiffusionPipeline.from_pretrained(
5 "stable-diffusion-v1-5/stable-diffusion-v1-5", torch_dtype=torch.float16).to("cuda")
6pipe.unet.requires_grad_(False) # the 2-D prior stays frozen
7alphas = pipe.scheduler.alphas_cumprod.to("cuda")
8
9def embed(text):
10 ids = pipe.tokenizer(text, padding="max_length", max_length=77, return_tensors="pt").input_ids
11 return pipe.text_encoder(ids.to("cuda"))[0]
12
13cond, uncond = embed("a DSLR photo of a corgi, side view"), embed("")
14
15def sds_loss(image, guidance=100.0): # image: (1, 3, 512, 512) in [0, 1]
16 latents = pipe.vae.encode(image.half() * 2 - 1).latent_dist.sample()
17 latents = latents * pipe.vae.config.scaling_factor
18 t = torch.randint(20, 981, (1,), device="cuda") # t/1000 in [0.02, 0.98]
19 noise = torch.randn_like(latents)
20 with torch.no_grad():
21 z_t = pipe.scheduler.add_noise(latents, noise, t)
22 eps = pipe.unet(torch.cat([z_t] * 2), t,
23 encoder_hidden_states=torch.cat([uncond, cond])).sample
24 eps_u, eps_c = eps.chunk(2)
25 eps_hat = eps_u + guidance * (eps_c - eps_u)
26 grad = (1 - alphas[t]) * (eps_hat - noise) # w(t) = sigma_t^2
27 return (grad.detach() * latents).sum() # d loss / d latents = grad
28
29loss = sds_loss(render(theta, camera)) # one random camera per step
30loss.backward()
31optimizer.step()
SDS loss for a differentiable render with Hugging Face diffusers, in latent space as in common open implementations (not run here: needs torch, diffusers and a GPU; render, theta and camera come from your NeRF or 3DGS code)

Choosing an approach

ApproachTime per assetStrengthsWeaknesses
SDS / VSD optimisationminutes to hoursno 3-D training data; any prompt the 2-D model knowsJanus faces, saturation, low diversity, slow
Multi-view diffusion + SDSminutesviews are consistent, far fewer Janus artefactsneeds 3-D renders to train the multi-view model
Multi-view diffusion + feed-forward reconstructionsecondsfast enough for interactive use; one network for all promptsdetail and topology limited by the reconstructor and its 3-D training set

Check yourself

You run SDS on the same prompt with 8 different seeds and get 8 nearly identical corgis. Why?

Check yourself

Why do multi-view diffusion models such as MVDream reduce the Janus (multi-face) problem when used as the SDS prior?

Remember

  • ✓SDS (DreamFusion, 2022): ∇θL=E[w(t)(ε^ϕ(zt;y,t)−ε) ∂x/∂θ]\nabla_\theta\mathcal L=\mathbb E[w(t)(\hat\varepsilon_\phi(z_t;y,t)-\varepsilon)\,\partial x/\partial\theta]. A frozen 2-D diffusion model critiques renders from random cameras; the U-Net Jacobian is skipped.
  • ✓DreamFusion needs guidance ω = 100, so its assets are saturated, smooth and similar across seeds. VSD (ProlificDreamer) works at ω = 7.5.
  • ✓The Janus problem comes from judging each view alone. Multi-view diffusion (Zero-1-to-3, MVDream) adds camera conditioning and cross-view attention.
  • ✓Feed-forward reconstructors (LRM, Instant3D) turn generated views into a 3-D asset in seconds instead of a per-prompt optimisation.
  • ✓Representations: NeRF in DreamFusion, 3-D Gaussians for speed (DreamGaussian), meshes for export; always inspect the exported geometry.
← back to Generative AI roadmap