Any-to-any models

Any-to-any models take and produce any mix of text, images, audio and video. Either everything becomes discrete tokens in one sequence (Chameleon, AnyGPT), or an LLM hub steers frozen encoders and diffusion decoders (NExT-GPT).

7 min read718 wordsadvanced0 viewsupdated 2026-10-21

Definition

Any-to-any model a generative model whose inputs and outputs can each be any combination of modalities, for example a photo and a spoken question in, an edited image and a spoken answer out. Representative work: CoDi, "Any-to-Any Generation via Composable Diffusion" (Tang et al., NeurIPS 2023); NExT-GPT (Wu, Fei, Qu, Ji and Chua, 2023); Unified-IO 2 (Lu et al., 2023); Chameleon, "Mixed-Modal Early-Fusion Foundation Models" (Chameleon Team, Meta, 2024); AnyGPT (Zhan et al., 2024).

Multimodal LLMs read images but answer in text. To get an image or speech back, a typical product chains models: a vision-language model, then a text-to-image model, then a TTS model, each with its own prompt. Every hand-off passes through text and loses information: the tone of a voice, the exact layout of the input image. Any-to-any models remove the hand-offs, so one model can keep a character consistent across a story with pictures, or answer speech with speech that carries emotion and timing.

There are two designs, and they differ in where pixels and waveforms are produced.

Two designs

(a) unified tokens / early fusion (Chameleon, AnyGPT)textBPE tokensimage 512²VQ: 32×32 codesspeechcodec 75 Hz × Kone transformernext token over thejoint vocabularytext tokens → textcodes → VQ decodercodes → codec dec.<s> draw a cat , then say hi <boi> 4012 77 931 … (1,024) <eoi> <boa> 311 8 … <eoa>every modality is generated token by token; no external generator, but images cost ~1,000 steps(b) LLM hub with adapters (NExT-GPT)imageaudio, videoImageBindfrozenin-projtrainedLLMVicuna + LoRAout-projtrainedStable DiffusionAudioLDMZeroscope videofrozen decoders"Here is a cat." [IMG1]…[IMGk]the LLM writes text plus a few signal tokens; their hidden states, projected, become thecondition of a diffusion decoder, so pixels and waveforms never pass through the LLM
fig.(a) Unified tokens: text through BPE, a 512² image through a VQ tokenizer into 32 × 32 = 1,024 codes, speech through a neural codec. One transformer predicts the next token over the joint vocabulary, and generated codes are decoded back to pixels or audio. (b) LLM hub: a frozen ImageBind encoder and a trained input projection feed the LLM, which writes text plus a few signal tokens; an output projection turns their hidden states into the conditioning of frozen Stable Diffusion, AudioLDM or Zeroscope decoders.

Unified tokens (early fusion). Each modality gets a discrete tokenizer: a Vector-quantized VAE (VQ-VAE)-style image tokenizer, a neural audio codec (see Neural codec models). Their codes are added to the vocabulary, and one decoder-only transformer is trained with the plain next-token loss on interleaved sequences. Chameleon (7B and 34B) does this for text and images; AnyGPT adds speech and music. Generation of any modality is ordinary sampling, so prompting, Constrained decoding and the KV cache apply unchanged.

LLM hub with adapters. Keep strong single-modality models frozen and connect them through the LLM. NExT-GPT uses ImageBind as a shared encoder, a projection into the LLM, and output projections from special signal tokens to the text conditioning of diffusion decoders. Only the projections and LoRA weights on the LLM are trained, around 1% of the parameters. Between the two sit hybrids such as Transfusion (Zhou et al., 2024), one transformer with a next-token loss on text and a diffusion loss on image patches.

Sequence budget

N=Ntext+∑images((rf)2+2)+∑clips(R T K+2),attention FLOPs∝N2N=N_\text{text}+\sum_\text{images}\Big(\big(\tfrac{r}{f}\big)^2+2\Big)+\sum_\text{clips}\big(R\,T\,K+2\big),\qquad \text{attention FLOPs}\propto N^2
r, fr,\ f
image side and tokenizer downsampling factor (512 and 16: 32 × 32 = 1,024 codes)
R, T, KR,\ T,\ K
codec frame rate, clip length in seconds, codebooks per frame (EnCodec 24 kHz: R = 75; K = 8 is its 6 kbps setting)
+2+2
begin / end markers such as <boi> and <eoi>
Bimage=(rf)2log⁡2∣C∣=1024×13=13,312 bits(∣C∣=8192)B_\text{image}=\big(\tfrac{r}{f}\big)^2\log_2|\mathcal C|=1024\times13=13{,}312\ \text{bits}\quad(|\mathcal C|=8192)
∣C∣|\mathcal C|
image codebook size; 8,192 in Chameleon

1,664 bytes for a 512 × 512 picture is 473× less than its 786,432 bytes of raw RGB. That compression is what makes images affordable in a sequence, and it is also the quality ceiling: fine text and small faces are lost in the tokenizer before the transformer sees them. In the hub design the same image is 32 learned queries on input (as in BLIP-2's Q-Former) and a few signal tokens on output; the diffusion decoder supplies the detail.

From the papers and model releases

Chameleon (2024)
image tokenizer: 512 × 512 → 1,024 tokens from an 8,192-code codebook; 7B and 34B models; reports that query-key normalisation was needed to keep mixed-modal training stable
NExT-GPT (2023)
ImageBind encoder, Vicuna LLM, Stable Diffusion / AudioLDM / Zeroscope decoders; projections plus LoRA trained (about 1% of parameters)
CoDi (2023)
separate diffusion models for each modality, aligned in a shared latent space so they can be composed for joint generation
Neural codecs
EnCodec at 24 kHz: 75 frames/s with 1,024-entry codebooks; 2 codebooks = 1.5 kbps, 8 = 6 kbps
GPT-4o (OpenAI, May 2024)
described as one network trained end to end across text, vision and audio; OpenAI quoted an average 320 ms audio response time at launch

Count the tokens of one exchange

one exchange: 60 text in, 1 image in → 40 text out, 1 image out, 5 s speech outunified tokens (early fusion): image = (512/16)² + 2 = 1026, speech = 75/s × 8 codebooks102610263002LLM hub + adapters: image in = 32 queries, each generated item = 4 signal tokens (same scale)the hub sequence zoomed ×3.3 to full width:60324044textimageaudiopale = input (prefill), solid = generatedunifiedhubsequence length N5154140tokens decoded one by one406848attention cost ∝ N² (relative)1355.3×1×external decoders run02text share of the sequence1.9%71.4%one VQ image at 512²: 1024 codes × 13 bits = 1664 bytes, 472.6× smaller than raw RGBspeech: 75 frames/s × 8 codebooks × 10 bits = 6.0 kbps, 600 tokens per secondthe hub is short because the diffusion decoders do the pixel and waveform work outside the LLM
image tokenizercodec
fig.Default exchange: 60 text tokens and one image in; 40 text tokens, one image and 5 s of speech out (512² images, 8 codebooks). Unified tokens: 5,154 positions, 4,068 of them generated one by one; text is 1.9% of the sequence. The hub: 140 positions, 48 decoded, plus 2 external decoder runs; attention over the unified sequence costs 1,355× more. Switch to 2 codebooks and the unified sequence drops to 2,904 (1,818 decoded). Drag speech to 30 s and it reaches 20,154. With no media at all both designs are the same 100 tokens.

Worked example: 10 s of speech is most of the context

PartFormulaTokensShare
Text60600.8%
Image(512/16)² + 21,02614.5%
Speech75 × 10 × 8 + 26,00284.7%
Total7,088100%
Same, 2 codebooks60 + 1,026 + (75 × 10 × 2 + 2)2,588
Same, 10 min of speech60 + 1,026 + (75 × 600 × 8 + 2)361,088
Unified tokens, 60 text tokens + one 512² image + 10 s of speech at 8 codebooks. Computed in Python

Audio, not images, is the expensive modality: 600 tokens per second at 8 codebooks, so ten minutes overflow a 128K-token context almost three times. Real systems cut this with fewer codebooks, semantic speech tokens at lower rates, or by predicting the codebooks of a frame in parallel rather than one by one. Compare with raw audio: 10 s at 24 kHz and 16 bits is 480,000 bytes, the 6 kbps codes are 7,500 bytes, 64× smaller.

python
1def unified_tokens(text=0, images=0, speech_s=0.0, res=512, codebooks=8, hz=75):
2 img = (res // 16) ** 2 + 2 # VQ codes + <boi>/<eoi>
3 aud = int(hz * speech_s * codebooks) + 2 if speech_s else 0
4 return text + images * img + aud
5
6print(unified_tokens(text=60, images=1, speech_s=10)) # 7088
7print(unified_tokens(text=60, images=1, speech_s=600)) # 361088: will not fit in 128K
8
9from openai import OpenAI
10resp = OpenAI().chat.completions.create(
11 model="gpt-4o-audio-preview",
12 modalities=["text", "audio"], # ask for both text and speech back
13 audio={"voice": "alloy", "format": "wav"},
14 messages=[{"role": "user", "content": "Say good morning in a cheerful voice."}],
15)
16wav_b64 = resp.choices[0].message.audio.data # base64 audio
17transcript = resp.choices[0].message.audio.transcript
A token-budget check for a unified-token model (run here: prints 7088 and 361088), then a text-plus-audio request in the OpenAI chat format as documented in late 2024 (not run here; model names and parameters change)

Use when

  • +Unified tokens: one model, one loss, true interleaved generation; reasoning can refer to earlier generated images
  • +Unified tokens: every LLM tool works on every modality (sampling, KV cache, constrained decoding)
  • +Hub: reuses strong frozen generators; cheap to train (projections + LoRA)
  • +Hub: short sequences, so long conversations with media stay affordable

Avoid when

  • −Unified tokens: quality capped by the tokenizer; ~1,000 sequential steps per image
  • −Unified tokens: audio inflates sequences (600 tokens/s at 8 codebooks)
  • −Hub: the LLM only steers through a few signal tokens, so fine control and consistency across outputs are weaker
  • −Hub: several models to serve, version and secure

Check yourself

In a unified-token model with a 512² VQ tokenizer (f = 16) and an 8-codebook 75 Hz codec, which adds more tokens: one generated image or 3 s of generated speech?

Check yourself

Why does NExT-GPT need to train only about 1% of its parameters?

Remember

  • ✓Any-to-any = any mix of modalities in and out, in one model rather than a chain of text hand-offs.
  • ✓Unified tokens (Chameleon, AnyGPT): VQ image codes and codec audio codes in one vocabulary, plain next-token training and sampling.
  • ✓LLM hub (NExT-GPT): frozen encoders and diffusion decoders, trained projections and signal tokens; about 1% of parameters trained.
  • ✓Budget: an image is (r/f)2+2(r/f)^2+2 tokens (1,026 at 512², f = 16); speech is R T KR\,T\,K (600 tokens/s at 75 Hz × 8): audio dominates.
  • ✓Unified models trade sequence length and tokenizer quality for true interleaving; hubs trade control for cheap training and short sequences.
← back to Generative AI roadmap