# Any-to-any models

> Any-to-any models take and produce any mix of text, images, audio and video. Either everything becomes discrete tokens in one sequence (Chameleon, AnyGPT), or an LLM hub steers frozen encoders and diffusion decoders (NExT-GPT).

Source: https://www.mlfoundry.in/learn/any-to-any-models  
Section: Generative AI › Multimodal Generative AI  
Level: advanced · 7 min read · updated 2026-10-21

**Any-to-any model:** a generative model whose inputs **and outputs** can each be any combination of modalities, for example a photo and a spoken question in, an edited image and a spoken answer out. Representative work: CoDi, "Any-to-Any Generation via Composable Diffusion" (Tang et al., NeurIPS 2023); NExT-GPT (Wu, Fei, Qu, Ji and Chua, 2023); Unified-IO 2 (Lu et al., 2023); Chameleon, "Mixed-Modal Early-Fusion Foundation Models" (Chameleon Team, Meta, 2024); AnyGPT (Zhan et al., 2024).

**Multimodal LLMs** read images but answer in text. To get an image or speech back, a typical product chains models: a vision-language model, then a text-to-image model, then a TTS model, each with its own prompt. Every hand-off passes through text and loses information: the tone of a voice, the exact layout of the input image. Any-to-any models remove the hand-offs, so one model can keep a character consistent across a story with pictures, or answer speech with speech that carries emotion and timing.

There are two designs, and they differ in where pixels and waveforms are produced.

## Two designs

> Figure: (a) Unified tokens: text through BPE, a 512² image through a VQ tokenizer into 32 × 32 = 1,024 codes, speech through a neural codec. One transformer predicts the next token over the joint vocabulary, and generated codes are decoded back to pixels or audio. (b) LLM hub: a frozen ImageBind encoder and a trained input projection feed the LLM, which writes text plus a few signal tokens; an output projection turns their hidden states into the conditioning of frozen Stable Diffusion, AudioLDM or Zeroscope decoders.

**Unified tokens (early fusion).** Each modality gets a discrete tokenizer: a **Vector-quantized VAE (VQ-VAE)**-style image tokenizer, a neural audio codec (see **Neural codec models**). Their codes are added to the vocabulary, and one decoder-only transformer is trained with the plain next-token loss on interleaved sequences. Chameleon (7B and 34B) does this for text and images; AnyGPT adds speech and music. Generation of any modality is ordinary sampling, so prompting, **Constrained decoding** and the **KV cache** apply unchanged.

**LLM hub with adapters.** Keep strong single-modality models frozen and connect them through the LLM. NExT-GPT uses ImageBind as a shared encoder, a projection into the LLM, and output projections from special signal tokens to the text conditioning of diffusion decoders. Only the projections and LoRA weights on the LLM are trained, around 1% of the parameters. Between the two sit hybrids such as Transfusion (Zhou et al., 2024), one transformer with a next-token loss on text and a diffusion loss on image patches.

## Sequence budget

$$
N=N_\text{text}+\sum_\text{images}\Big(\big(\tfrac{r}{f}\big)^2+2\Big)+\sum_\text{clips}\big(R\,T\,K+2\big),\qquad \text{attention FLOPs}\propto N^2
$$

- $r,\ f$: image side and tokenizer downsampling factor (512 and 16: 32 × 32 = 1,024 codes)
- $R,\ T,\ K$: codec frame rate, clip length in seconds, codebooks per frame (EnCodec 24 kHz: R = 75; K = 8 is its 6 kbps setting)
- $+2$: begin / end markers such as <boi> and <eoi>

$$
B_\text{image}=\big(\tfrac{r}{f}\big)^2\log_2|\mathcal C|=1024\times13=13{,}312\ \text{bits}\quad(|\mathcal C|=8192)
$$

- $|\mathcal C|$: image codebook size; 8,192 in Chameleon

1,664 bytes for a 512 × 512 picture is 473× less than its 786,432 bytes of raw RGB. That compression is what makes images affordable in a sequence, and it is also the quality ceiling: fine text and small faces are lost in the tokenizer before the transformer sees them. In the hub design the same image is 32 learned queries on input (as in BLIP-2's Q-Former) and a few signal tokens on output; the diffusion decoder supplies the detail.

**From the papers and model releases**

- **Chameleon (2024):** image tokenizer: 512 × 512 → 1,024 tokens from an 8,192-code codebook; 7B and 34B models; reports that query-key normalisation was needed to keep mixed-modal training stable
- **NExT-GPT (2023):** ImageBind encoder, Vicuna LLM, Stable Diffusion / AudioLDM / Zeroscope decoders; projections plus LoRA trained (about 1% of parameters)
- **CoDi (2023):** separate diffusion models for each modality, aligned in a shared latent space so they can be composed for joint generation
- **Neural codecs:** EnCodec at 24 kHz: 75 frames/s with 1,024-entry codebooks; 2 codebooks = 1.5 kbps, 8 = 6 kbps
- **GPT-4o (OpenAI, May 2024):** described as one network trained end to end across text, vision and audio; OpenAI quoted an average 320 ms audio response time at launch

## Count the tokens of one exchange

> Figure: Default exchange: 60 text tokens and one image in; 40 text tokens, one image and 5 s of speech out (512² images, 8 codebooks). Unified tokens: 5,154 positions, 4,068 of them generated one by one; text is 1.9% of the sequence. The hub: 140 positions, 48 decoded, plus 2 external decoder runs; attention over the unified sequence costs 1,355× more. Switch to 2 codebooks and the unified sequence drops to 2,904 (1,818 decoded). Drag speech to 30 s and it reaches 20,154. With no media at all both designs are the same 100 tokens.

## Worked example: 10 s of speech is most of the context

Unified tokens, 60 text tokens + one 512² image + 10 s of speech at 8 codebooks. Computed in Python

| Part | Formula | Tokens | Share |
| --- | --- | --- | --- |
| Text | 60 | 60 | 0.8% |
| Image | (512/16)² + 2 | 1,026 | 14.5% |
| Speech | 75 × 10 × 8 + 2 | 6,002 | 84.7% |
| Total |  | 7,088 | 100% |
| Same, 2 codebooks | 60 + 1,026 + (75 × 10 × 2 + 2) | 2,588 |  |
| Same, 10 min of speech | 60 + 1,026 + (75 × 600 × 8 + 2) | 361,088 |  |

Audio, not images, is the expensive modality: 600 tokens per second at 8 codebooks, so ten minutes overflow a 128K-token context almost three times. Real systems cut this with fewer codebooks, semantic speech tokens at lower rates, or by predicting the codebooks of a frame in parallel rather than one by one. Compare with raw audio: 10 s at 24 kHz and 16 bits is 480,000 bytes, the 6 kbps codes are 7,500 bytes, 64× smaller.

```python
def unified_tokens(text=0, images=0, speech_s=0.0, res=512, codebooks=8, hz=75):
    img = (res // 16) ** 2 + 2                          # VQ codes + <boi>/<eoi>
    aud = int(hz * speech_s * codebooks) + 2 if speech_s else 0
    return text + images * img + aud

print(unified_tokens(text=60, images=1, speech_s=10))      # 7088
print(unified_tokens(text=60, images=1, speech_s=600))     # 361088: will not fit in 128K

from openai import OpenAI
resp = OpenAI().chat.completions.create(
    model="gpt-4o-audio-preview",
    modalities=["text", "audio"],                     # ask for both text and speech back
    audio={"voice": "alloy", "format": "wav"},
    messages=[{"role": "user", "content": "Say good morning in a cheerful voice."}],
)
wav_b64 = resp.choices[0].message.audio.data        # base64 audio
transcript = resp.choices[0].message.audio.transcript
```

A token-budget check for a unified-token model (run here: prints 7088 and 361088), then a text-plus-audio request in the OpenAI chat format as documented in late 2024 (not run here; model names and parameters change)

**Strengths**

- Unified tokens: one model, one loss, true interleaved generation; reasoning can refer to earlier generated images
- Unified tokens: every LLM tool works on every modality (sampling, KV cache, constrained decoding)
- Hub: reuses strong frozen generators; cheap to train (projections + LoRA)
- Hub: short sequences, so long conversations with media stay affordable

**Limitations**

- Unified tokens: quality capped by the tokenizer; ~1,000 sequential steps per image
- Unified tokens: audio inflates sequences (600 tokens/s at 8 codebooks)
- Hub: the LLM only steers through a few signal tokens, so fine control and consistency across outputs are weaker
- Hub: several models to serve, version and secure

> **Choosing for a product:** If outputs are mostly text with an occasional picture, a hub (or a plain chain of models) is cheaper and easier to upgrade one part at a time. If the product depends on interleaving, such as illustrated stories with a consistent character or speech-to-speech with low latency and preserved tone, a unified model avoids the information lost at each text hand-off. Measure the token budget first: it decides latency and cost more than the parameter count does.

> **Safety filters must cover every output channel:** A text-only moderation step does not see the generated image or audio. Any-to-any outputs need filters per modality (image classifiers, speech transcription plus text moderation) and provenance marking for generated media; see **Content moderation** and **Watermarking generated content**. Voice output also raises impersonation risk, which is why released systems restrict voices to preset ones.

**Question:** In a unified-token model with a 512² VQ tokenizer (f = 16) and an 8-codebook 75 Hz codec, which adds more tokens: one generated image or 3 s of generated speech?

- A. The image: 1,026 tokens against 1,802
- B. The speech: 1,802 tokens against 1,026
- C. They are equal
- D. The image, because pixels carry more information

**Answer:** B. The speech: 1,802 tokens against 1,026 — Image: $(512/16)^2+2=1{,}026$. Speech: $75\times3\times8+2=1{,}802$. Audio at a high codebook count overtakes images after under 2 seconds, which is why speech tokenizers with fewer tokens per second matter for these models.

**Question:** Why does NExT-GPT need to train only about 1% of its parameters?

- A. It uses a smaller LLM
- B. The encoders, the LLM backbone and the diffusion decoders are pretrained and frozen; only the input/output projections and LoRA are trained to connect them
- C. It generates images as text descriptions
- D. It trains each modality separately with no shared model

**Answer:** B. The encoders, the LLM backbone and the diffusion decoders are pretrained and frozen; only the input/output projections and LoRA are trained to connect them — The hub design aligns representations instead of learning to generate pixels: projections map ImageBind features into the LLM and map signal-token states into each decoder's conditioning space.

**Key points**

- Any-to-any = any mix of modalities in and out, in one model rather than a chain of text hand-offs.
- Unified tokens (Chameleon, AnyGPT): VQ image codes and codec audio codes in one vocabulary, plain next-token training and sampling.
- LLM hub (NExT-GPT): frozen encoders and diffusion decoders, trained projections and signal tokens; about 1% of parameters trained.
- Budget: an image is $(r/f)^2+2$ tokens (1,026 at 512², f = 16); speech is $R\,T\,K$ (600 tokens/s at 75 Hz × 8): audio dominates.
- Unified models trade sequence length and tokenizer quality for true interleaving; hubs trade control for cheap training and short sequences.

---

From MLFoundry (https://www.mlfoundry.in). The web page adds interactive figures and diagrams.
