Definition
Text-to-3D generation produces a 3-D asset (a radiance field, a set of 3-D Gaussians or a textured mesh) from a text prompt. The founding method is DreamFusion (Poole, Jain, Barron and Mildenhall, Google, 2022), which introduced score distillation sampling (SDS): optimise a 3-D scene so that its renders from random cameras look plausible to a frozen text-to-image diffusion model.
The obstacle is data. Text–image pairs number in the billions. Captioned 3-D assets number in the hundreds of thousands to millions: Objaverse (Deitke et al., 2023) has over 800K objects and Objaverse-XL over 10M, of very mixed quality. Point-E (Nichol et al., OpenAI, 2022) trained directly on 3-D data and produced point clouds in 1–2 minutes on one GPU, but its quality and prompt range lagged image models.
DreamFusion's answer was to need no 3-D data at all. A 2-D model already knows what a corgi looks like from the side and from above. If every render of a 3-D scene looks like a corgi from that angle, the scene is a 3-D corgi. The scene representation is one of the DL articles: Neural radiance fields (NeRF) in DreamFusion, 3D Gaussian splatting in faster successors such as DreamGaussian (2023), usually converted to a mesh at the end.
Score distillation sampling
Render an image of scene parameters from camera . Noise it as the diffusion model was trained to see, ask the frozen model to predict the noise, and push the render in the direction that makes the prediction match. DreamFusion showed that the gradient of the diffusion loss with the U-Net Jacobian dropped is a well-behaved update:
- noise predicted by the frozen diffusion model for prompt ; no gradient flows through it
- the Gaussian noise that was actually added
- timestep weighting; is a common choice
- backpropagation through the differentiable renderer only
- classifier-free guidance scale. Image sampling uses about 7.5; DreamFusion used
- empty prompt (unconditional prediction)
SDS is mode-seeking. It minimises a KL divergence between a Gaussian around the render and the model's noisy distribution, so it climbs towards one high-density image rather than sampling. Large sharpens the target so that the gradients from different views agree. That is why DreamFusion needs 100. The side effects are well documented: over-saturated colours, over-smooth surfaces and little diversity across seeds. ProlificDreamer (Wang et al., 2023) replaced SDS with variational score distillation. It swaps the true noise for the prediction of a LoRA-tuned copy of the model that tracks the current renders, and works at ordinary guidance (7.5).
Reported numbers (from the papers)
- DreamFusion (2022)
- frozen Imagen 64×64 base model, mip-NeRF 360-style scene, ω = 100, t ~ U(0.02, 0.98); about 1.5 h per prompt on a 4-chip TPUv4 machine
- Magic3D (2023)
- coarse neural field, then a textured mesh refined with a latent diffusion model at 512×512; about 40 min, reported as 2× faster than DreamFusion
- Janus problem
- SDS often grows a front face on every side, because each view is judged alone and front views dominate image data; view-dependent prompts only partly fix it
- Zero-1-to-3 (2023)
- Stable Diffusion fine-tuned on Objaverse renders to synthesise a new view of an object given the relative camera rotation and translation
- MVDream (2023)
- generates 4 views jointly, with self-attention across the views and camera embeddings; as an SDS prior it largely removes Janus artefacts
- LRM (2023)
- a transformer of about 500M parameters, trained on about 1M objects, maps one image to a triplane NeRF in about 5 s
- Instant3D (2023)
- text → 2×2 grid of 4 views (fine-tuned diffusion) → sparse-view reconstructor; about 20 s per asset
Worked example: one SDS step in one dimension
| Step | Formula | Value |
|---|---|---|
| noisy render | z = αx + σε = 0.16 + 0.30 | 0.460 |
| noisy prior variance | v = α²s² + σ² | 0.3664 |
| prior score | (αμ − z) / v | 0.4913 |
| predicted noise | ε̂ = −σ · score | −0.2948 |
| residual | ε̂ − ε | −0.7948 |
| SDS gradient | w · (ε̂ − ε) | −0.2861 |
| update | x − 0.5 · gradient | 0.343 (towards 0.8) |
Averaged over , the gradient is . It always points at the mode, and the single-sample value is close. The noise term has coefficient here. Once the expected gradient is zero, whatever is. That is the collapse the viz shows: the spread of the prior never enters the fixed point.
1import torch2from diffusers import StableDiffusionPipeline34pipe = StableDiffusionPipeline.from_pretrained(5 "stable-diffusion-v1-5/stable-diffusion-v1-5", torch_dtype=torch.float16).to("cuda")6pipe.unet.requires_grad_(False) # the 2-D prior stays frozen7alphas = pipe.scheduler.alphas_cumprod.to("cuda")89def embed(text):10 ids = pipe.tokenizer(text, padding="max_length", max_length=77, return_tensors="pt").input_ids11 return pipe.text_encoder(ids.to("cuda"))[0]1213cond, uncond = embed("a DSLR photo of a corgi, side view"), embed("")1415def sds_loss(image, guidance=100.0): # image: (1, 3, 512, 512) in [0, 1]16 latents = pipe.vae.encode(image.half() * 2 - 1).latent_dist.sample()17 latents = latents * pipe.vae.config.scaling_factor18 t = torch.randint(20, 981, (1,), device="cuda") # t/1000 in [0.02, 0.98]19 noise = torch.randn_like(latents)20 with torch.no_grad():21 z_t = pipe.scheduler.add_noise(latents, noise, t)22 eps = pipe.unet(torch.cat([z_t] * 2), t,23 encoder_hidden_states=torch.cat([uncond, cond])).sample24 eps_u, eps_c = eps.chunk(2)25 eps_hat = eps_u + guidance * (eps_c - eps_u)26 grad = (1 - alphas[t]) * (eps_hat - noise) # w(t) = sigma_t^227 return (grad.detach() * latents).sum() # d loss / d latents = grad2829loss = sds_loss(render(theta, camera)) # one random camera per step30loss.backward()31optimizer.step()Choosing an approach
| Approach | Time per asset | Strengths | Weaknesses |
|---|---|---|---|
| SDS / VSD optimisation | minutes to hours | no 3-D training data; any prompt the 2-D model knows | Janus faces, saturation, low diversity, slow |
| Multi-view diffusion + SDS | minutes | views are consistent, far fewer Janus artefacts | needs 3-D renders to train the multi-view model |
| Multi-view diffusion + feed-forward reconstruction | seconds | fast enough for interactive use; one network for all prompts | detail and topology limited by the reconstructor and its 3-D training set |
Check yourself
You run SDS on the same prompt with 8 different seeds and get 8 nearly identical corgis. Why?
Check yourself
Why do multi-view diffusion models such as MVDream reduce the Janus (multi-face) problem when used as the SDS prior?
Remember
- ✓SDS (DreamFusion, 2022): . A frozen 2-D diffusion model critiques renders from random cameras; the U-Net Jacobian is skipped.
- ✓DreamFusion needs guidance ω = 100, so its assets are saturated, smooth and similar across seeds. VSD (ProlificDreamer) works at ω = 7.5.
- ✓The Janus problem comes from judging each view alone. Multi-view diffusion (Zero-1-to-3, MVDream) adds camera conditioning and cross-view attention.
- ✓Feed-forward reconstructors (LRM, Instant3D) turn generated views into a 3-D asset in seconds instead of a per-prompt optimisation.
- ✓Representations: NeRF in DreamFusion, 3-D Gaussians for speed (DreamGaussian), meshes for export; always inspect the exported geometry.