# Text-to-video generation

> Text-to-video models are latent diffusion over spacetime: a 3-D VAE compresses the clip, spacetime patches become tokens and temporal attention keeps frames consistent. Tokens grow with duration × resolution, full attention with their square.

Source: https://www.mlfoundry.in/learn/text-to-video-generation  
Section: Generative AI › Multimodal Generative AI  
Level: advanced · 8 min read · updated 2026-10-21

**Text-to-video generation:** produces a short clip, a tensor of $T$ frames $\times H\times W\times 3$, from a text prompt. Current systems are **diffusion models** over a compressed spacetime latent. Key steps: Video Diffusion Models (Ho et al., 2022), Make-A-Video (Singer et al., Meta, 2022), Imagen Video (Ho et al., Google, 2022), Video LDM, "Align your Latents" (Blattmann et al., 2023), and OpenAI's Sora technical report, "Video generation models as world simulators" (February 2024).

A video model has to do what a **Text-to-image generation** model does for every frame, and also keep identity, lighting and motion coherent across frames. Generating each frame independently from the same prompt gives a slideshow: the object changes shape, the background flickers, nothing moves smoothly.

There is also much less training data. Paired image–text data runs to billions of examples, while captioned video is scarcer and noisier. Most designs therefore reuse a pretrained image model for appearance and learn motion from video, often from video that has no captions at all. Make-A-Video was trained without any paired text–video data.

## From pixels to spacetime tokens

Diffusing raw pixels is too expensive, so the clip is first compressed, as in **Latent diffusion models**. Early systems ran an image VAE on each frame. Newer ones use a 3-D (spatio-temporal) VAE. A common choice, for example CogVideoX (2024), downsamples 4× in time and 8×8 in space into 16 channels. The VAE is causal in time: the first frame is encoded on its own, so $1+4k$ frames become $1+k$ latent frames, and the first frame doubles as an image.

$$
T'=1+\frac{T-1}{f_t},\qquad S=\frac{H}{f_s\,p}\cdot\frac{W}{f_s\,p},\qquad N=T'\cdot S
$$

- $T,\,H,\,W$: frames and pixel resolution of the clip
- $f_t,\,f_s$: VAE downsampling in time (e.g. 4) and in space (e.g. 8)
- $p$: spatial patch size of the transformer (e.g. 2, so each token covers $16\times16$ pixels of one latent frame)
- $N$: number of spacetime tokens the denoiser sees

The Sora report calls these units **spacetime patches** and treats them like the tokens of an LLM. Because a video of any length or aspect ratio is just a different number of patches, one model can be trained on videos at their native size and duration instead of cropping everything to a fixed square. The denoiser is a diffusion transformer (DiT, Peebles and Xie, 2023), trained with the usual noise-prediction loss on the latent.

$$
\mathcal L=\mathbb E_{z_0,\,\varepsilon,\,t}\big\|\varepsilon-\varepsilon_\theta(\alpha_t z_0+\sigma_t\varepsilon,\;t,\;c)\big\|^2,\qquad z_0\in\mathbb R^{T'\times h\times w\times 16}
$$

- $z_0$: VAE latent of the whole clip; all frames are noised and denoised together
- $c$: text-encoder output for the caption, read through cross-attention
- $\alpha_t,\,\sigma_t$: noise schedule at step t

> Figure: A 5-second 720p clip at 24 fps is 121 frames. The causal 3-D VAE (4× in time, 8×8 in space) turns it into 31 latent frames of 90×160×16, about 47× fewer numbers. Patching 2×2 gives 45×80 = 3,600 tokens per latent frame and 111,600 tokens in total. Each denoiser block mixes them within a frame (spatial attention), along time at each position (temporal attention), and with the prompt (cross-attention). Sampling runs the denoiser about 50 times from pure noise, then the decoder expands 31 latent frames back to 121.

## Temporal layers and attention cost

Video Diffusion Models used **factorised** space–time attention: spatial attention inside each frame, then temporal attention across frames at each spatial position. Video LDM and AnimateDiff (Guo et al., 2023) add such temporal layers to a pretrained text-to-image model. The image layers stay frozen and only the new layers train on video. Video LDM blends each temporal layer's output with the spatial one through a learned mixing factor. AnimateDiff zero-initialises the motion module's output projection, so at step 0 the model is exactly the image model applied to each frame, and motion is learned on top.

$$
\text{full: }N^2=(T'S)^2\qquad\text{factorised: }T'S^2+S\,T'^2
$$

- $N^2$: query–key pairs per attention layer when every token attends to every other (Sora-style DiT, CogVideoX)
- $T'S^2$: spatial attention: S tokens attend to each other in each of T′ frames
- $S\,T'^2$: temporal attention: T′ tokens attend to each other at each of S positions

Full 3-D attention lets any patch look at any other, which helps large motion and keeping objects consistent when they leave and re-enter the frame. It costs a factor of about $T'$ more than factorised attention. It is only practical with memory-efficient kernels such as **FlashAttention**, which never materialise the $N\times N$ score matrix.

**Reported systems (from the papers)**

- **Video Diffusion Models (2022):** 3-D U-Net with factorised space–time attention; trained jointly on images and videos
- **Make-A-Video (2022):** appearance from text–image pairs, motion from unlabelled video; no paired text–video data
- **Imagen Video (2022):** cascade of 7 diffusion models (a base model plus spatial and temporal super-resolution) reaching 1280×768 at 24 fps
- **Stable Video Diffusion (2023):** latent video model by Stability AI generating 14 or 25 frames at 576×1024; the paper stresses data curation
- **Sora report (Feb 2024):** spacetime patches, diffusion transformer, native aspect ratios, up to a minute of HD video; re-captioned training videos; sample quality improved with training compute
- **FVD (Unterthiner et al., 2018):** Fréchet distance between I3D video features of real and generated clips, the video analogue of FID

> Figure: Default: 5 s at 24 fps, 720×1280, 3-D VAE. That is 121 frames, 31 latent frames, 3,600 tokens per frame and N = 111,600. Full attention is 1.25e10 pairs per layer and factorised attention 4.05e8, 30.7× fewer. One fp16 score matrix for one head would be 24.9 GB. Drag to 20 s: N = 435,600, 1.90e11 pairs (379.5 GB per head), and the gap is 117.1×. Switching to a per-frame image VAE at 5 s gives the same 435,600 tokens, because temporal compression alone cuts N by about 4×. At 1 s (25 frames, 7 latent frames) the gap is only 7.0×.

## Worked example: a 49-frame clip

6 s at 8 fps, 480×720 (the clip size of the open CogVideoX models), VAE 4× / 8×8, patch 2; computed in Python

| Quantity | Formula | Value |
| --- | --- | --- |
| frames T | 6·8 + 1 | 49 |
| latent frames T′ | 1 + 48/4 | 13 |
| latent size | 480/8 × 720/8 × 16 | 60 × 90 × 16 |
| compression | 49·480·720·3 / (13·60·90·16) | 45.2× |
| tokens per frame S | (60/2)·(90/2) | 1,350 |
| tokens N | 13 · 1,350 | 17,550 |
| full attention pairs | 17,550² | 3.08e8 |
| factorised pairs | 13·1,350² + 1,350·13² | 2.39e7 (12.9× fewer) |

Doubling the duration doubles $T'$. Full attention then costs about 4× more. Factorised attention costs about 2× more, because its spatial term $T'S^2$ dominates. Doubling the resolution in both directions multiplies $S$ by 4, so full attention costs 16× more.

```python
import torch
from diffusers import CogVideoXPipeline
from diffusers.utils import export_to_video

pipe = CogVideoXPipeline.from_pretrained("THUDM/CogVideoX-2b", torch_dtype=torch.float16)
pipe.enable_model_cpu_offload()     # keep only the active sub-model on the GPU
pipe.vae.enable_tiling()            # decode the 3-D latent in tiles to save memory

video = pipe(
    prompt="A red kite flying over a windy beach at sunset, slow pan to the left",
    num_frames=49,                  # 1 + 4k frames for the causal 3-D VAE
    num_inference_steps=50,
    guidance_scale=6.0,             # classifier-free guidance
    generator=torch.Generator("cuda").manual_seed(0),
).frames[0]
export_to_video(video, "kite.mp4", fps=8)
```

Generating a 49-frame clip with an open model via Hugging Face diffusers (not run here: needs torch, diffusers and a GPU)

## Getting coherent clips in practice

**Common practitioner workflow**

1. Write a **descriptive** prompt: subject, action, camera motion, lighting. Models trained on re-captioned data (dense, long captions, as in the Sora report) follow detailed prompts better; many products expand short prompts with an LLM first.
2. Prefer **image-to-video** when the look matters: generate or pick a keyframe with a text-to-image model, then animate it. Appearance is then fixed and the video model only has to supply motion.
3. Generate at the model's trained frame count and resolution. Off-size requests often degrade motion, because temporal layers saw a fixed $T'$ range.
4. For longer clips, extend autoregressively from the last frames, or generate keyframes and interpolate. Expect drift: errors accumulate with each extension.
5. Upscale or interpolate frames in a separate pass (Imagen Video's cascade does this inside the model).

> **Per-frame metrics miss flicker:** Frame-wise FID or **CLIP score** can look excellent on a clip that flickers, because each frame alone is a good image. Measure temporal quality directly: FVD on I3D features, similarity of consecutive frames, and human ratings of motion. Then check the failure modes the Sora report lists: physics such as glass shattering, objects that appear or vanish, and cause and effect that does not carry through.

> **Why reuse a frozen image model:** The spatial layers already encode appearance learned from billions of image–text pairs. Training only the temporal layers needs far less video. AnimateDiff's motion module can also be plugged into personalised image checkpoints (for example a **DreamBooth** fine-tune) without retraining them.

**Strengths**

- Spacetime tokens let one transformer handle any duration, resolution and aspect ratio
- Inflating a pretrained image model reuses its appearance prior; only motion must be learned
- A 3-D VAE gives about 4× fewer tokens than a per-frame VAE at the same resolution

**Limitations**

- Full attention cost grows with (duration × resolution)²; long clips need memory-efficient kernels and many GPUs
- Sampling runs a large transformer ~50 times over 10⁴–10⁵ tokens: seconds of video take minutes of GPU time
- Physics, object permanence and long-range consistency remain weak points; artefacts are easier to spot in motion than in a still

**Question:** A 720p clip goes from 5 s to 10 s at the same fps and VAE. Roughly how much does attention cost per layer grow, full vs factorised?

- A. 2× and 2×
- B. ≈4× and ≈2×
- C. ≈4× and ≈4×
- D. ≈2× and ≈4×

**Answer:** B. ≈4× and ≈2× — $T'$ goes from 31 to 61. Full attention scales with $T'^2$: $4.82\times10^{10}/1.25\times10^{10}\approx3.9$. Factorised attention is dominated by the spatial term $T'S^2$, which is linear in $T'$: $8.04\times10^8/4.05\times10^8\approx2.0$.

**Question:** AnimateDiff inserts a motion module with a zero-initialised output projection into a frozen Stable Diffusion and trains only that module. What does the network compute at the start of training?

- A. Random noise, because the temporal layers are random
- B. The original image model applied to each frame independently
- C. A copy of the first frame repeated T times
- D. An average of all frames

**Answer:** B. The original image model applied to each frame independently — A zero-initialised output projection means the motion module adds nothing at first, so the model starts as the image model on each frame, which already makes good stills, and training adds the cross-frame mixing that turns them into motion.

**Key points**

- Tokens: $N=T'S$ with $T'=1+(T-1)/f_t$ and $S=(H/f_sp)(W/f_sp)$. A 5 s 720p clip at 24 fps with a 4×/8×8 VAE and 2×2 patches is 111,600 tokens.
- Full 3-D attention costs $N^2$, factorised costs $T'S^2+ST'^2$, about $T'$ times less. Sora-style DiTs pay for full attention with efficient kernels.
- Temporal layers added to a frozen text-to-image model (Video LDM, AnimateDiff) reuse the image prior and learn only motion.
- Spacetime patches (Sora report, 2024) let one model train on and generate any duration and aspect ratio.
- Judge temporal quality with FVD, frame-to-frame consistency and human review, not per-frame FID; expect physics and permanence failures.

---

From MLFoundry (https://www.mlfoundry.in). The web page adds interactive figures and diagrams.
