Text-to-video generation

Text-to-video models are latent diffusion over spacetime: a 3-D VAE compresses the clip, spacetime patches become tokens and temporal attention keeps frames consistent. Tokens grow with duration × resolution, full attention with their square.

8 min read844 wordsadvanced0 viewsupdated 2026-10-21

Definition

Text-to-video generation produces a short clip, a tensor of TT frames ×H×W×3\times H\times W\times 3, from a text prompt. Current systems are diffusion models over a compressed spacetime latent. Key steps: Video Diffusion Models (Ho et al., 2022), Make-A-Video (Singer et al., Meta, 2022), Imagen Video (Ho et al., Google, 2022), Video LDM, "Align your Latents" (Blattmann et al., 2023), and OpenAI's Sora technical report, "Video generation models as world simulators" (February 2024).

A video model has to do what a Text-to-image generation model does for every frame, and also keep identity, lighting and motion coherent across frames. Generating each frame independently from the same prompt gives a slideshow: the object changes shape, the background flickers, nothing moves smoothly.

There is also much less training data. Paired image–text data runs to billions of examples, while captioned video is scarcer and noisier. Most designs therefore reuse a pretrained image model for appearance and learn motion from video, often from video that has no captions at all. Make-A-Video was trained without any paired text–video data.

From pixels to spacetime tokens

Diffusing raw pixels is too expensive, so the clip is first compressed, as in Latent diffusion models. Early systems ran an image VAE on each frame. Newer ones use a 3-D (spatio-temporal) VAE. A common choice, for example CogVideoX (2024), downsamples 4× in time and 8×8 in space into 16 channels. The VAE is causal in time: the first frame is encoded on its own, so 1+4k1+4k frames become 1+k1+k latent frames, and the first frame doubles as an image.

T′=1+T−1ft,S=Hfs p⋅Wfs p,N=T′⋅ST'=1+\frac{T-1}{f_t},\qquad S=\frac{H}{f_s\,p}\cdot\frac{W}{f_s\,p},\qquad N=T'\cdot S
T, H, WT,\,H,\,W
frames and pixel resolution of the clip
ft, fsf_t,\,f_s
VAE downsampling in time (e.g. 4) and in space (e.g. 8)
pp
spatial patch size of the transformer (e.g. 2, so each token covers 16×1616\times16 pixels of one latent frame)
NN
number of spacetime tokens the denoiser sees

The Sora report calls these units spacetime patches and treats them like the tokens of an LLM. Because a video of any length or aspect ratio is just a different number of patches, one model can be trained on videos at their native size and duration instead of cropping everything to a fixed square. The denoiser is a diffusion transformer (DiT, Peebles and Xie, 2023), trained with the usual noise-prediction loss on the latent.

L=Ez0, ε, t∥ε−εθ(αtz0+σtε,  t,  c)∥2,z0∈RT′×h×w×16\mathcal L=\mathbb E_{z_0,\,\varepsilon,\,t}\big\|\varepsilon-\varepsilon_\theta(\alpha_t z_0+\sigma_t\varepsilon,\;t,\;c)\big\|^2,\qquad z_0\in\mathbb R^{T'\times h\times w\times 16}
z0z_0
VAE latent of the whole clip; all frames are noised and denoised together
cc
text-encoder output for the caption, read through cross-attention
αt, σt\alpha_t,\,\sigma_t
noise schedule at step t
latent video diffusion: 5 s at 24 fps, 720p → 111,600 tokensvideo clip121×720×1280×33-D causal VAE4× time, 8×8 spacelatent z₀31×90×160×16spacetime patches2×2 → 31×45×80N = 111,600 tokenstrain: z_t = α_t·z₀ + σ_t·εprompt“a red kite overa beach, slowpan left”text encoder(e.g. T5)denoiser transformer, × L blocksspatial attn: within a frame, S = 3,600temporal attn: same patch, T′ = 31cross-attn: tokens → text embeddingsMLP, conditioned on timestep tsamplingz_T ~ N(0, I)repeat ~50×:ε_θ(z_t, t, text)z_t → z_t−1+ classifier-freeguidanceVAE decoder31 → 121 frames7 of 121 decoded frames (5.04 s at 24 fps)Sora report (2024): spacetime patches into a diffusion transformer; variable duration, aspect ratioVideo LDM, AnimateDiff (2023): temporal layers added to a frozen text-to-image model
fig.A 5-second 720p clip at 24 fps is 121 frames. The causal 3-D VAE (4× in time, 8×8 in space) turns it into 31 latent frames of 90×160×16, about 47× fewer numbers. Patching 2×2 gives 45×80 = 3,600 tokens per latent frame and 111,600 tokens in total. Each denoiser block mixes them within a frame (spatial attention), along time at each position (temporal attention), and with the prompt (cross-attention). Sampling runs the denoiser about 50 times from pure noise, then the decoder expands 31 latent frames back to 121.

Temporal layers and attention cost

Video Diffusion Models used factorised space–time attention: spatial attention inside each frame, then temporal attention across frames at each spatial position. Video LDM and AnimateDiff (Guo et al., 2023) add such temporal layers to a pretrained text-to-image model. The image layers stay frozen and only the new layers train on video. Video LDM blends each temporal layer's output with the spatial one through a learned mixing factor. AnimateDiff zero-initialises the motion module's output projection, so at step 0 the model is exactly the image model applied to each frame, and motion is learned on top.

full: N2=(T′S)2factorised: T′S2+S T′2\text{full: }N^2=(T'S)^2\qquad\text{factorised: }T'S^2+S\,T'^2
N2N^2
query–key pairs per attention layer when every token attends to every other (Sora-style DiT, CogVideoX)
T′S2T'S^2
spatial attention: S tokens attend to each other in each of T′ frames
S T′2S\,T'^2
temporal attention: T′ tokens attend to each other at each of S positions

Full 3-D attention lets any patch look at any other, which helps large motion and keeping objects consistent when they leave and re-enter the frame. It costs a factor of about T′T' more than factorised attention. It is only practical with memory-efficient kernels such as FlashAttention, which never materialise the N×NN\times N score matrix.

Reported systems (from the papers)

Video Diffusion Models (2022)
3-D U-Net with factorised space–time attention; trained jointly on images and videos
Make-A-Video (2022)
appearance from text–image pairs, motion from unlabelled video; no paired text–video data
Imagen Video (2022)
cascade of 7 diffusion models (a base model plus spatial and temporal super-resolution) reaching 1280×768 at 24 fps
Stable Video Diffusion (2023)
latent video model by Stability AI generating 14 or 25 frames at 576×1024; the paper stresses data curation
Sora report (Feb 2024)
spacetime patches, diffusion transformer, native aspect ratios, up to a minute of HD video; re-captioned training videos; sample quality improved with training compute
FVD (Unterthiner et al., 2018)
Fréchet distance between I3D video features of real and generated clips, the video analogue of FID
5 s at 24 fps, 720×1280 · VAE 4× time, 8×8 space · patch 2×2schematic: 5 latent frames × 6×6 spacetime patches; outlined = query tokent′ = 1t′ = 2t′ = 3t′ = 4t′ = 5full 3-D attentionquery sees 180 of 180 tokensevery patch, every framecost ∝ N² = (T′S)²factorised = sum of bothattention pairs per layer vs clip length (log scale)1e51e71e91e111e131s5s10s15s20sfull N²factorised T′S² + ST′²frames121latent frames T'31tokens / frame S3,600tokens N = T'·S111,600full pairs N²1.25e10T'S² + ST'²4.05e8full / factorised30.7×fp16 N² per head24.9 GBmaterialised scores would not fit: memory-efficient kernels (FlashAttention) are mandatory
attention
resolution
fps · VAE
fig.Default: 5 s at 24 fps, 720×1280, 3-D VAE. That is 121 frames, 31 latent frames, 3,600 tokens per frame and N = 111,600. Full attention is 1.25e10 pairs per layer and factorised attention 4.05e8, 30.7× fewer. One fp16 score matrix for one head would be 24.9 GB. Drag to 20 s: N = 435,600, 1.90e11 pairs (379.5 GB per head), and the gap is 117.1×. Switching to a per-frame image VAE at 5 s gives the same 435,600 tokens, because temporal compression alone cuts N by about 4×. At 1 s (25 frames, 7 latent frames) the gap is only 7.0×.

Worked example: a 49-frame clip

QuantityFormulaValue
frames T6·8 + 149
latent frames T′1 + 48/413
latent size480/8 × 720/8 × 1660 × 90 × 16
compression49·480·720·3 / (13·60·90·16)45.2×
tokens per frame S(60/2)·(90/2)1,350
tokens N13 · 1,35017,550
full attention pairs17,550²3.08e8
factorised pairs13·1,350² + 1,350·13²2.39e7 (12.9× fewer)
6 s at 8 fps, 480×720 (the clip size of the open CogVideoX models), VAE 4× / 8×8, patch 2; computed in Python

Doubling the duration doubles T′T'. Full attention then costs about 4× more. Factorised attention costs about 2× more, because its spatial term T′S2T'S^2 dominates. Doubling the resolution in both directions multiplies SS by 4, so full attention costs 16× more.

python
1import torch
2from diffusers import CogVideoXPipeline
3from diffusers.utils import export_to_video
4
5pipe = CogVideoXPipeline.from_pretrained("THUDM/CogVideoX-2b", torch_dtype=torch.float16)
6pipe.enable_model_cpu_offload() # keep only the active sub-model on the GPU
7pipe.vae.enable_tiling() # decode the 3-D latent in tiles to save memory
8
9video = pipe(
10 prompt="A red kite flying over a windy beach at sunset, slow pan to the left",
11 num_frames=49, # 1 + 4k frames for the causal 3-D VAE
12 num_inference_steps=50,
13 guidance_scale=6.0, # classifier-free guidance
14 generator=torch.Generator("cuda").manual_seed(0),
15).frames[0]
16export_to_video(video, "kite.mp4", fps=8)
Generating a 49-frame clip with an open model via Hugging Face diffusers (not run here: needs torch, diffusers and a GPU)

Getting coherent clips in practice

Common practitioner workflow

  1. 1Write a descriptive prompt: subject, action, camera motion, lighting. Models trained on re-captioned data (dense, long captions, as in the Sora report) follow detailed prompts better; many products expand short prompts with an LLM first.
  2. 2Prefer image-to-video when the look matters: generate or pick a keyframe with a text-to-image model, then animate it. Appearance is then fixed and the video model only has to supply motion.
  3. 3Generate at the model's trained frame count and resolution. Off-size requests often degrade motion, because temporal layers saw a fixed T′T' range.
  4. 4For longer clips, extend autoregressively from the last frames, or generate keyframes and interpolate. Expect drift: errors accumulate with each extension.
  5. 5Upscale or interpolate frames in a separate pass (Imagen Video's cascade does this inside the model).

Use when

  • +Spacetime tokens let one transformer handle any duration, resolution and aspect ratio
  • +Inflating a pretrained image model reuses its appearance prior; only motion must be learned
  • +A 3-D VAE gives about 4× fewer tokens than a per-frame VAE at the same resolution

Avoid when

  • −Full attention cost grows with (duration × resolution)²; long clips need memory-efficient kernels and many GPUs
  • −Sampling runs a large transformer ~50 times over 10⁴–10⁵ tokens: seconds of video take minutes of GPU time
  • −Physics, object permanence and long-range consistency remain weak points; artefacts are easier to spot in motion than in a still

Check yourself

A 720p clip goes from 5 s to 10 s at the same fps and VAE. Roughly how much does attention cost per layer grow, full vs factorised?

Check yourself

AnimateDiff inserts a motion module with a zero-initialised output projection into a frozen Stable Diffusion and trains only that module. What does the network compute at the start of training?

Remember

  • ✓Tokens: N=T′SN=T'S with T′=1+(T−1)/ftT'=1+(T-1)/f_t and S=(H/fsp)(W/fsp)S=(H/f_sp)(W/f_sp). A 5 s 720p clip at 24 fps with a 4×/8×8 VAE and 2×2 patches is 111,600 tokens.
  • ✓Full 3-D attention costs N2N^2, factorised costs T′S2+ST′2T'S^2+ST'^2, about T′T' times less. Sora-style DiTs pay for full attention with efficient kernels.
  • ✓Temporal layers added to a frozen text-to-image model (Video LDM, AnimateDiff) reuse the image prior and learn only motion.
  • ✓Spacetime patches (Sora report, 2024) let one model train on and generate any duration and aspect ratio.
  • ✓Judge temporal quality with FVD, frame-to-frame consistency and human review, not per-frame FID; expect physics and permanence failures.
← back to Generative AI roadmap