# Text-to-3D generation

> Text-to-3D either distils a frozen 2-D diffusion model into a NeRF or Gaussian scene with score distillation sampling (DreamFusion, 2022), or generates consistent multi-view images and reconstructs them feed-forward in seconds.

Source: https://www.mlfoundry.in/learn/text-to-3d-generation  
Section: Generative AI › Multimodal Generative AI  
Level: advanced · 8 min read · updated 2026-10-21

**Text-to-3D generation:** produces a 3-D asset (a radiance field, a set of 3-D Gaussians or a textured mesh) from a text prompt. The founding method is DreamFusion (Poole, Jain, Barron and Mildenhall, Google, 2022), which introduced **score distillation sampling (SDS)**: optimise a 3-D scene so that its renders from random cameras look plausible to a frozen text-to-image diffusion model.

The obstacle is data. Text–image pairs number in the billions. Captioned 3-D assets number in the hundreds of thousands to millions: Objaverse (Deitke et al., 2023) has over 800K objects and Objaverse-XL over 10M, of very mixed quality. Point-E (Nichol et al., OpenAI, 2022) trained directly on 3-D data and produced point clouds in 1–2 minutes on one GPU, but its quality and prompt range lagged image models.

DreamFusion's answer was to need no 3-D data at all. A 2-D model already knows what a corgi looks like from the side and from above. If every render of a 3-D scene looks like a corgi from that angle, the scene is a 3-D corgi. The scene representation is one of the DL articles: **Neural radiance fields (NeRF)** in DreamFusion, **3D Gaussian splatting** in faster successors such as DreamGaussian (2023), usually converted to a mesh at the end.

## Score distillation sampling

Render an image $x=g(\theta,c)$ of scene parameters $\theta$ from camera $c$. Noise it as the diffusion model was trained to see, ask the frozen model to predict the noise, and push the render in the direction that makes the prediction match. DreamFusion showed that the gradient of the diffusion loss with the U-Net Jacobian dropped is a well-behaved update:

$$
\nabla_\theta\mathcal L_{\text{SDS}}=\mathbb E_{t,\varepsilon,c}\Big[w(t)\,\big(\hat\varepsilon_\phi(z_t;\,y,\,t)-\varepsilon\big)\,\frac{\partial x}{\partial\theta}\Big],\qquad z_t=\alpha_t\,x+\sigma_t\,\varepsilon
$$

- $\hat\varepsilon_\phi$: noise predicted by the frozen diffusion model $\phi$ for prompt $y$; no gradient flows through it
- $\varepsilon$: the Gaussian noise that was actually added
- $w(t)$: timestep weighting; $\sigma_t^2=1-\bar\alpha_t$ is a common choice
- $\partial x/\partial\theta$: backpropagation through the differentiable renderer only

$$
\hat\varepsilon_\phi(z_t;y,t)=\varepsilon_\phi(z_t;\varnothing,t)+\omega\,\big(\varepsilon_\phi(z_t;y,t)-\varepsilon_\phi(z_t;\varnothing,t)\big)
$$

- $\omega$: classifier-free guidance scale. Image sampling uses about 7.5; DreamFusion used $\omega=100$
- $\varnothing$: empty prompt (unconditional prediction)

SDS is mode-seeking. It minimises a KL divergence between a Gaussian around the render and the model's noisy distribution, so it climbs towards one high-density image rather than sampling. Large $\omega$ sharpens the target so that the gradients from different views agree. That is why DreamFusion needs 100. The side effects are well documented: over-saturated colours, over-smooth surfaces and little diversity across seeds. ProlificDreamer (Wang et al., 2023) replaced SDS with **variational score distillation**. It swaps the true noise $\varepsilon$ for the prediction of a LoRA-tuned copy of the model that tracks the current renders, and works at ordinary guidance (7.5).

> Figure: One SDS step: pick a random camera, render (DreamFusion renders 64×64 for the Imagen base model), add noise at a random t in (0.02, 0.98), and let the frozen model predict the noise for a view-dependent prompt ("…, side view"). The weighted residual (ε_φ − ε) is backpropagated through the renderer into θ. Thousands of such steps make up one asset. Below are the three families in use: per-prompt SDS optimisation (minutes to hours), multi-view diffusion models, and feed-forward reconstructors that turn a few views into a 3-D asset in seconds.

**Reported numbers (from the papers)**

- **DreamFusion (2022):** frozen Imagen 64×64 base model, mip-NeRF 360-style scene, ω = 100, t ~ U(0.02, 0.98); about 1.5 h per prompt on a 4-chip TPUv4 machine
- **Magic3D (2023):** coarse neural field, then a textured mesh refined with a latent diffusion model at 512×512; about 40 min, reported as 2× faster than DreamFusion
- **Janus problem:** SDS often grows a front face on every side, because each view is judged alone and front views dominate image data; view-dependent prompts only partly fix it
- **Zero-1-to-3 (2023):** Stable Diffusion fine-tuned on Objaverse renders to synthesise a new view of an object given the relative camera rotation and translation
- **MVDream (2023):** generates 4 views jointly, with self-attention across the views and camera embeddings; as an SDS prior it largely removes Janus artefacts
- **LRM (2023):** a transformer of about 500M parameters, trained on about 1M objects, maps one image to a triplane NeRF in about 5 s
- **Instant3D (2023):** text → 2×2 grid of 4 views (fine-tuned diffusion) → sparse-view reconstructor; about 20 s per asset

> Figure: SDS on a toy 2-D "image space" with an exact mixture prior (cat, dog, car; prompt weights 0.90/0.05/0.05). Each of 16 random initialisations is one asset. At ω = 1, after 400 steps all 16 runs sit within 0.6 of the cat mode, with mean (−0.90, 0.44), 0.120 from the mode centre. Their spread is 0.019, against 0.186 for 16 true samples: SDS collapses to a mode, it does not sample. At ω = 2 they still collapse (spread 0.018) but the mean moves 0.371 out from the mode, away from the unconditional mean. At ω = 7.5 none are within 0.6 (the mean is 1.837 away), and at ω = 100 every run leaves the chart (9.226 away). This is the toy analogue of over-saturated colours.

## Worked example: one SDS step in one dimension

Prior for the prompt: N(μ = 0.8, s = 0.1); render x = 0.20; α = 0.8, σ = 0.6, sampled ε = 0.5, w = σ², ω = 1, learning rate 0.5 (computed in Python)

| Step | Formula | Value |
| --- | --- | --- |
| noisy render | z = αx + σε = 0.16 + 0.30 | 0.460 |
| noisy prior variance | v = α²s² + σ² | 0.3664 |
| prior score | (αμ − z) / v | 0.4913 |
| predicted noise | ε̂ = −σ · score | −0.2948 |
| residual | ε̂ − ε | −0.7948 |
| SDS gradient | w · (ε̂ − ε) | −0.2861 |
| update | x − 0.5 · gradient | 0.343 (towards 0.8) |

Averaged over $\varepsilon$, the gradient is $-w\,\sigma\alpha(\mu-x)/v=-0.2830$. It always points at the mode, and the single-sample value $-0.2861$ is close. The noise term $\varepsilon(\sigma^2/v-1)$ has coefficient $-0.017$ here. Once $x=\mu$ the expected gradient is zero, whatever $s$ is. That is the collapse the viz shows: the spread of the prior never enters the fixed point.

```python
import torch
from diffusers import StableDiffusionPipeline

pipe = StableDiffusionPipeline.from_pretrained(
    "stable-diffusion-v1-5/stable-diffusion-v1-5", torch_dtype=torch.float16).to("cuda")
pipe.unet.requires_grad_(False)                    # the 2-D prior stays frozen
alphas = pipe.scheduler.alphas_cumprod.to("cuda")

def embed(text):
    ids = pipe.tokenizer(text, padding="max_length", max_length=77, return_tensors="pt").input_ids
    return pipe.text_encoder(ids.to("cuda"))[0]

cond, uncond = embed("a DSLR photo of a corgi, side view"), embed("")

def sds_loss(image, guidance=100.0):               # image: (1, 3, 512, 512) in [0, 1]
    latents = pipe.vae.encode(image.half() * 2 - 1).latent_dist.sample()
    latents = latents * pipe.vae.config.scaling_factor
    t = torch.randint(20, 981, (1,), device="cuda") # t/1000 in [0.02, 0.98]
    noise = torch.randn_like(latents)
    with torch.no_grad():
        z_t = pipe.scheduler.add_noise(latents, noise, t)
        eps = pipe.unet(torch.cat([z_t] * 2), t,
                        encoder_hidden_states=torch.cat([uncond, cond])).sample
        eps_u, eps_c = eps.chunk(2)
        eps_hat = eps_u + guidance * (eps_c - eps_u)
    grad = (1 - alphas[t]) * (eps_hat - noise)     # w(t) = sigma_t^2
    return (grad.detach() * latents).sum()         # d loss / d latents = grad

loss = sds_loss(render(theta, camera))             # one random camera per step
loss.backward()
optimizer.step()
```

SDS loss for a differentiable render with Hugging Face diffusers, in latent space as in common open implementations (not run here: needs torch, diffusers and a GPU; render, theta and camera come from your NeRF or 3DGS code)

## Choosing an approach

| Approach | Time per asset | Strengths | Weaknesses |
| --- | --- | --- | --- |
| SDS / VSD optimisation | minutes to hours | no 3-D training data; any prompt the 2-D model knows | Janus faces, saturation, low diversity, slow |
| Multi-view diffusion + SDS | minutes | views are consistent, far fewer Janus artefacts | needs 3-D renders to train the multi-view model |
| Multi-view diffusion + feed-forward reconstruction | seconds | fast enough for interactive use; one network for all prompts | detail and topology limited by the reconstructor and its 3-D training set |

> **Check the asset, not the turntable:** A rendered turntable can hide what a downstream user needs: holes, floating density, back-faces with a second face, lighting baked into the texture. Export the mesh, inspect normals and UVs, and render it under neutral light in your target engine. For evaluation, view-averaged **CLIP score** measures prompt fidelity but says little about geometry.

> **Why the Jacobian can be dropped:** The U-Net Jacobian $\partial\hat\varepsilon/\partial z_t$ is expensive to backpropagate and, as DreamFusion found, poorly conditioned at small noise levels. Without it, the gradient of the diffusion loss is just the residual $(\hat\varepsilon-\varepsilon)$ pushed through the renderer. The 2-D model is used only as a critic, never trained, so any off-the-shelf checkpoint, including a **DreamBooth** fine-tune, can serve as the prior.

**Question:** You run SDS on the same prompt with 8 different seeds and get 8 nearly identical corgis. Why?

- A. The renderer ignores the seed
- B. SDS is mode-seeking: it drives each render to a high-density image of the guided distribution instead of sampling from it
- C. Classifier-free guidance adds noise
- D. The text encoder is deterministic

**Answer:** B. SDS is mode-seeking: it drives each render to a high-density image of the guided distribution instead of sampling from it — The expected SDS gradient vanishes at a mode of the (guided, noise-smoothed) distribution, regardless of how spread out that distribution is. High $\omega$ makes the target even sharper. Variational score distillation (ProlificDreamer) was designed to restore sample diversity.

**Question:** Why do multi-view diffusion models such as MVDream reduce the Janus (multi-face) problem when used as the SDS prior?

- A. They render at higher resolution
- B. They judge several camera-conditioned views jointly, with attention across views, so a face on the back conflicts with the face on the front
- C. They use a larger guidance scale
- D. They skip the noise step

**Answer:** B. They judge several camera-conditioned views jointly, with attention across views, so a face on the back conflicts with the face on the front — A single-view 2-D prior scores each render alone, and a front-facing face looks plausible from every angle. A multi-view model is trained on consistent renders of 3-D objects and denoises the views together with known cameras, so inconsistent views are improbable.

**Key points**

- SDS (DreamFusion, 2022): $\nabla_\theta\mathcal L=\mathbb E[w(t)(\hat\varepsilon_\phi(z_t;y,t)-\varepsilon)\,\partial x/\partial\theta]$. A frozen 2-D diffusion model critiques renders from random cameras; the U-Net Jacobian is skipped.
- DreamFusion needs guidance ω = 100, so its assets are saturated, smooth and similar across seeds. VSD (ProlificDreamer) works at ω = 7.5.
- The Janus problem comes from judging each view alone. Multi-view diffusion (Zero-1-to-3, MVDream) adds camera conditioning and cross-view attention.
- Feed-forward reconstructors (LRM, Instant3D) turn generated views into a 3-D asset in seconds instead of a per-prompt optimisation.
- Representations: NeRF in DreamFusion, 3-D Gaussians for speed (DreamGaussian), meshes for export; always inspect the exported geometry.

---

From MLFoundry (https://www.mlfoundry.in). The web page adds interactive figures and diagrams.
