Definition
Text-to-video generation produces a short clip, a tensor of frames , from a text prompt. Current systems are diffusion models over a compressed spacetime latent. Key steps: Video Diffusion Models (Ho et al., 2022), Make-A-Video (Singer et al., Meta, 2022), Imagen Video (Ho et al., Google, 2022), Video LDM, "Align your Latents" (Blattmann et al., 2023), and OpenAI's Sora technical report, "Video generation models as world simulators" (February 2024).
A video model has to do what a Text-to-image generation model does for every frame, and also keep identity, lighting and motion coherent across frames. Generating each frame independently from the same prompt gives a slideshow: the object changes shape, the background flickers, nothing moves smoothly.
There is also much less training data. Paired image–text data runs to billions of examples, while captioned video is scarcer and noisier. Most designs therefore reuse a pretrained image model for appearance and learn motion from video, often from video that has no captions at all. Make-A-Video was trained without any paired text–video data.
From pixels to spacetime tokens
Diffusing raw pixels is too expensive, so the clip is first compressed, as in Latent diffusion models. Early systems ran an image VAE on each frame. Newer ones use a 3-D (spatio-temporal) VAE. A common choice, for example CogVideoX (2024), downsamples 4× in time and 8×8 in space into 16 channels. The VAE is causal in time: the first frame is encoded on its own, so frames become latent frames, and the first frame doubles as an image.
- frames and pixel resolution of the clip
- VAE downsampling in time (e.g. 4) and in space (e.g. 8)
- spatial patch size of the transformer (e.g. 2, so each token covers pixels of one latent frame)
- number of spacetime tokens the denoiser sees
The Sora report calls these units spacetime patches and treats them like the tokens of an LLM. Because a video of any length or aspect ratio is just a different number of patches, one model can be trained on videos at their native size and duration instead of cropping everything to a fixed square. The denoiser is a diffusion transformer (DiT, Peebles and Xie, 2023), trained with the usual noise-prediction loss on the latent.
- VAE latent of the whole clip; all frames are noised and denoised together
- text-encoder output for the caption, read through cross-attention
- noise schedule at step t
Temporal layers and attention cost
Video Diffusion Models used factorised space–time attention: spatial attention inside each frame, then temporal attention across frames at each spatial position. Video LDM and AnimateDiff (Guo et al., 2023) add such temporal layers to a pretrained text-to-image model. The image layers stay frozen and only the new layers train on video. Video LDM blends each temporal layer's output with the spatial one through a learned mixing factor. AnimateDiff zero-initialises the motion module's output projection, so at step 0 the model is exactly the image model applied to each frame, and motion is learned on top.
- query–key pairs per attention layer when every token attends to every other (Sora-style DiT, CogVideoX)
- spatial attention: S tokens attend to each other in each of T′ frames
- temporal attention: T′ tokens attend to each other at each of S positions
Full 3-D attention lets any patch look at any other, which helps large motion and keeping objects consistent when they leave and re-enter the frame. It costs a factor of about more than factorised attention. It is only practical with memory-efficient kernels such as FlashAttention, which never materialise the score matrix.
Reported systems (from the papers)
- Video Diffusion Models (2022)
- 3-D U-Net with factorised space–time attention; trained jointly on images and videos
- Make-A-Video (2022)
- appearance from text–image pairs, motion from unlabelled video; no paired text–video data
- Imagen Video (2022)
- cascade of 7 diffusion models (a base model plus spatial and temporal super-resolution) reaching 1280×768 at 24 fps
- Stable Video Diffusion (2023)
- latent video model by Stability AI generating 14 or 25 frames at 576×1024; the paper stresses data curation
- Sora report (Feb 2024)
- spacetime patches, diffusion transformer, native aspect ratios, up to a minute of HD video; re-captioned training videos; sample quality improved with training compute
- FVD (Unterthiner et al., 2018)
- Fréchet distance between I3D video features of real and generated clips, the video analogue of FID
Worked example: a 49-frame clip
| Quantity | Formula | Value |
|---|---|---|
| frames T | 6·8 + 1 | 49 |
| latent frames T′ | 1 + 48/4 | 13 |
| latent size | 480/8 × 720/8 × 16 | 60 × 90 × 16 |
| compression | 49·480·720·3 / (13·60·90·16) | 45.2× |
| tokens per frame S | (60/2)·(90/2) | 1,350 |
| tokens N | 13 · 1,350 | 17,550 |
| full attention pairs | 17,550² | 3.08e8 |
| factorised pairs | 13·1,350² + 1,350·13² | 2.39e7 (12.9× fewer) |
Doubling the duration doubles . Full attention then costs about 4× more. Factorised attention costs about 2× more, because its spatial term dominates. Doubling the resolution in both directions multiplies by 4, so full attention costs 16× more.
1import torch2from diffusers import CogVideoXPipeline3from diffusers.utils import export_to_video45pipe = CogVideoXPipeline.from_pretrained("THUDM/CogVideoX-2b", torch_dtype=torch.float16)6pipe.enable_model_cpu_offload() # keep only the active sub-model on the GPU7pipe.vae.enable_tiling() # decode the 3-D latent in tiles to save memory89video = pipe(10 prompt="A red kite flying over a windy beach at sunset, slow pan to the left",11 num_frames=49, # 1 + 4k frames for the causal 3-D VAE12 num_inference_steps=50,13 guidance_scale=6.0, # classifier-free guidance14 generator=torch.Generator("cuda").manual_seed(0),15).frames[0]16export_to_video(video, "kite.mp4", fps=8)Getting coherent clips in practice
Common practitioner workflow
- 1Write a descriptive prompt: subject, action, camera motion, lighting. Models trained on re-captioned data (dense, long captions, as in the Sora report) follow detailed prompts better; many products expand short prompts with an LLM first.
- 2Prefer image-to-video when the look matters: generate or pick a keyframe with a text-to-image model, then animate it. Appearance is then fixed and the video model only has to supply motion.
- 3Generate at the model's trained frame count and resolution. Off-size requests often degrade motion, because temporal layers saw a fixed range.
- 4For longer clips, extend autoregressively from the last frames, or generate keyframes and interpolate. Expect drift: errors accumulate with each extension.
- 5Upscale or interpolate frames in a separate pass (Imagen Video's cascade does this inside the model).
Use when
- +Spacetime tokens let one transformer handle any duration, resolution and aspect ratio
- +Inflating a pretrained image model reuses its appearance prior; only motion must be learned
- +A 3-D VAE gives about 4× fewer tokens than a per-frame VAE at the same resolution
Avoid when
- −Full attention cost grows with (duration × resolution)²; long clips need memory-efficient kernels and many GPUs
- −Sampling runs a large transformer ~50 times over 10⁴–10⁵ tokens: seconds of video take minutes of GPU time
- −Physics, object permanence and long-range consistency remain weak points; artefacts are easier to spot in motion than in a still
Check yourself
A 720p clip goes from 5 s to 10 s at the same fps and VAE. Roughly how much does attention cost per layer grow, full vs factorised?
Check yourself
AnimateDiff inserts a motion module with a zero-initialised output projection into a frozen Stable Diffusion and trains only that module. What does the network compute at the start of training?
Remember
- ✓Tokens: with and . A 5 s 720p clip at 24 fps with a 4×/8×8 VAE and 2×2 patches is 111,600 tokens.
- ✓Full 3-D attention costs , factorised costs , about times less. Sora-style DiTs pay for full attention with efficient kernels.
- ✓Temporal layers added to a frozen text-to-image model (Video LDM, AnimateDiff) reuse the image prior and learn only motion.
- ✓Spacetime patches (Sora report, 2024) let one model train on and generate any duration and aspect ratio.
- ✓Judge temporal quality with FVD, frame-to-frame consistency and human review, not per-frame FID; expect physics and permanence failures.