# Qwen

> Qwen (Alibaba, 2023–2025) is a Llama-style decoder family with one ~152K byte-level BPE vocabulary, QKV bias until Qwen3 replaced it with QK-Norm, GQA from Qwen2, RoPE base 10⁴→10⁶ plus YaRN and DCA for 128K context, MoE variants, and 2.4T→36T training tokens.

Source: https://www.mlfoundry.in/learn/qwen  
Section: Deep Learning › Large Language Models (LLMs)  
Level: advanced · 15 min read · updated 2026-10-13

**Qwen:** (通义千问, Tongyi Qianwen) is the open-weight language-model series of Alibaba Cloud's Qwen team. It is documented in Bai et al., "Qwen Technical Report" (2023), Yang et al., "Qwen2 Technical Report" (2024), the Qwen team's "Qwen2.5 Technical Report" (2024), and Yang et al., "Qwen3 Technical Report" (2025). Qwen1.5 (February 2024) was released with a blog post rather than a paper. Every generation is a **Decoder-only transformer** in the **Llama** mould: pre-RMSNorm, SwiGLU, RoPE and an untied head. All share one byte-level BPE tokenizer of about 152K tokens. The generations differ in a few block details (QKV bias, then QK-Norm), attention layout (MHA, then GQA), long-context method, mixture-of-experts variants and, above all, the amount of data: about 2–3T tokens for Qwen, 7T for Qwen2, 18T for Qwen2.5 and 36T for Qwen3.

Qwen began in 2023 as Alibaba's answer to LLaMA. It aimed at strong English *and* Chinese, plus code and maths, with open weights. LLaMA's 32K SentencePiece vocabulary covers Chinese poorly: a sentence breaks into many byte-fallback pieces, which wastes context and compute. Qwen therefore started from a large BPE vocabulary and kept it fixed for later generations, while scaling everything else. Later releases grew to 119 languages and dialects (Qwen3), contexts of 128K tokens and more, a 235B mixture-of-experts model, and a single model that can either reason step by step or answer directly.

## The block and what changed

> Figure: The Qwen block, redrawn for each generation; hover or click a generation on the right, and the parts that changed from the previous one get an orange outline. Qwen (2023) is LLaMA's block with full multi-head attention, biases on the q, k and v projections, and RoPE with base 10,000. Qwen1.5 adds GQA only in the 32B and 110B models and a first MoE model (60 routed experts, top-4, plus a shared expert). Qwen2 uses GQA at every size, raises the RoPE base to 1,000,000 and adds a 64-expert MoE. Qwen2.5 keeps the architecture and scales the data to 18T tokens. Qwen3 drops the QKV bias in favour of QK-Norm and uses 128-expert MoE layers with no shared expert. The tokenizer is the same throughout.

$$
\textbf{Qwen–Qwen2.5: }\ q=W_qx+b_q,\ k=W_kx+b_k,\ v=W_vx+b_v\qquad\quad \textbf{Qwen3: }\ q=\mathrm{RMSNorm}_q(W_qx),\ k=\mathrm{RMSNorm}_k(W_kx),\ v=W_vx
$$

- $b_q,b_k,b_v$: the only biases in the model. The Qwen report adds them "to enhance the extrapolation ability" of RoPE; for Qwen2.5-7B they are 129,024 parameters, 0.0017% of the model
- $\mathrm{RMSNorm}_q,\ \mathrm{RMSNorm}_k$: QK-Norm: RMSNorm over each head's $d_h=128$ coordinates, applied before RoPE, with one learned gain vector shared by all heads
- $x$: the pre-RMSNorm-ed residual stream
- $\text{QK-Norm}$: bounds the size of $q^\top k$, so attention logits cannot blow up in bf16. Qwen3 adopts it "to ensure stable training" (the idea is from Dehghani et al.'s ViT-22B, 2023)

The QKV bias is a small detail with a practical consequence. Any code that re-implements the block from a plain Llama definition will silently drop $b_q,b_k,b_v$ and get wrong outputs. This is why Hugging Face has separate `Qwen2` and `Qwen3` model classes, although both are otherwise Llama-shaped.

## The tokenizer

Qwen's tokenizer starts from the `cl100k_base` byte-level BPE of OpenAI's tiktoken (see **Byte-pair encoding (BPE)**) and adds frequently used Chinese characters and words, plus tokens for other languages. Numbers are split into single digits. Byte-level means any UTF-8 string is representable, so there is never an unknown token. The Qwen2 report gives the vocabulary as 151,643 regular tokens plus 3 control tokens. Qwen2.5 expands the control tokens to 22, and Qwen3 adds `<think>`-style tokens for reasoning. Checkpoints pad the embedding to 151,936 or 152,064 rows. The Qwen report showed that this vocabulary compresses Chinese, and most of the other languages it tested, better than LLaMA's 32K tokenizer, which means more text per context window and per FLOP.

$$
\text{embedding share}=\frac{V\,d\,(2-\mathbb 1_{\text{tied}})}{N}:\qquad \text{Qwen2.5-0.5B: }\frac{151936\cdot896}{494{,}032{,}768}=27.6\%,\qquad \text{Qwen2.5-7B: }\frac{2\cdot152064\cdot3584}{7{,}615{,}616{,}512}=14.3\%
$$

- $V$: embedding rows in the checkpoint (padded vocabulary)
- $\mathbb 1_{\text{tied}}$: 1 when the head reuses the embedding; Qwen ties it only in the smallest models (Qwen2.5 0.5B–3B, Qwen3 0.6B–4B)
- $N$: total parameters

A 152K vocabulary is expensive for small models. Untied, Qwen2.5-0.5B would spend 43% of its parameters on the two tables. Tying, and calling the rest "non-embedding parameters" (0.36B), is how the model cards present it.

## Long context: RoPE base, YaRN and dual chunk attention

Qwen (2023) trained at 2048 tokens and extended at inference with training-free tricks. It used dynamic NTK-aware interpolation (see **Rotary positional embeddings (RoPE)**), LogN attention scaling, which multiplies the logits by $\log_{L_{train}}n$ when the length $n$ exceeds the training length, and layer-wise window attention. From Qwen2 on, the recipe has two steps. First, the final pre-training stage raises the context to 32K and the RoPE base from 10,000 to 1,000,000 (adjusted base frequency, ABF), so that low-frequency pairs rotate slowly enough to keep long distances apart. Second, inference uses YaRN with factor 4 to reach about 128K, together with Dual Chunk Attention (DCA; An et al., "Training-Free Long-Context Scaling of Large Language Models", 2024). DCA splits a long sequence into chunks and remaps relative positions so that no query–key distance exceeds what the model saw in training.

$$
\theta_i=b^{-2i/d_h},\qquad \theta'_i=\big(1-\gamma(r_i)\big)\frac{\theta_i}{s}+\gamma(r_i)\,\theta_i,\qquad r_i=\frac{L_{orig}\,\theta_i}{2\pi},\qquad \text{logits}\times\big(0.1\ln s+1\big)^2
$$

- $b$: RoPE base: $10^4$ in Qwen, $10^6$ from Qwen2 (raised during long-context pre-training)
- $s=4,\ L_{orig}=32768$: Qwen2.5 and Qwen3 model cards: `rope_scaling = {"type": "yarn", "factor": 4.0, "original_max_position_embeddings": 32768}`
- $\gamma(r)$: YaRN's ramp: 0 for pairs making fewer than 1 turn within $L_{orig}$ (fully interpolated), 1 above 32 turns (kept), linear between
- $(0.1\ln4+1)^2$: $=1.296$, YaRN's attention temperature at $s=4$

Qwen2.5-Turbo and the open Qwen2.5-7B/14B-Instruct-1M models (early 2025) pushed this to 1M tokens. They used progressive long-context training plus DCA and sparse attention at inference.

**Generations**

- **Qwen (2023):** 1.8B, 7B, 14B, 72B. 7B: $L=32$, $d=4096$, 32 heads (MHA), $d_{ff}=11008$; 14B: 40 layers, $d=5120$, 40 heads. Up to 3T tokens (2.4T for 7B), context 2048 in pre-training. Chat models via SFT and RLHF, and code and maths specialists (Code-Qwen, Math-Qwen)
- **Qwen1.5 (Feb 2024):** 0.5B to 110B dense, 32K context for all sizes; GQA only in 32B and 110B. Qwen1.5-MoE-A2.7B: 14.3B total, 2.7B active, 60 routed experts (top-4) plus a shared expert as wide as four of them
- **Qwen2 (Jun 2024):** 0.5B, 1.5B, 7B, 72B and 57B-A14B (64 experts, top-8, plus shared). GQA at every size; 7T tokens (12T for 0.5B); about 30 languages. 72B: $L=80$, $d=8192$, 64 query / 8 KV heads, $d_{ff}=29568$
- **Qwen2.5 (Sep 2024):** 0.5B, 1.5B, 3B, 7B, 14B, 32B, 72B; 18T tokens. Same architecture as Qwen2 (Qwen2.5-7B: 28 layers, $d=3584$, 28/4 heads, 7.61B parameters). 128K context with YaRN and up to 8K generated tokens; Qwen2.5-Turbo and -Plus are API-only MoE models
- **Qwen3 (Apr 2025):** dense 0.6B, 1.7B, 4B, 8B, 14B, 32B; MoE 30B-A3B and 235B-A22B (128 experts, 8 active, no shared expert, global-batch load-balancing loss). 36T tokens in 119 languages and dialects; QK-Norm, no QKV bias. Apache 2.0
- **Qwen3 pre-training:** three stages: general data at 4K sequence length (over 30T tokens); a knowledge- and reasoning-heavy stage of about 5T tokens; then long-context training at 32K with the RoPE base raised from 10,000 to 1,000,000
- **Qwen3-235B-A22B:** $L=94$, $d=4096$, 64 query / 4 KV heads of 128, experts of width 1536; 235B total, 22B active per token

## Mixture of experts in Qwen

Each Qwen MoE layer replaces the dense SwiGLU with $E$ small SwiGLU experts and a linear router (see **Mixture of experts (MoE)** and **Sparse MoE routing**). The router picks the top-$k$ experts per token and mixes their outputs with softmax weights. Qwen1.5-MoE and Qwen2-MoE also have an always-on *shared expert* whose output passes through a sigmoid gate. Its job is to hold common knowledge so that the routed experts can specialise. Qwen3 drops the shared expert and balances the experts with a load-balancing loss computed over the whole global batch rather than per micro-batch, which allows experts to specialise by domain.

> Figure: Exact parameter counts for eleven released checkpoints, computed from their configs. The totals match the model cards: Qwen2.5-7B 7.61B (6.53B non-embedding), Qwen3-8B 8.19B, Qwen3-30B-A3B 30.5B with 3.3B active, and Qwen3-235B-A22B 235B with 22B active. The default, Qwen3-30B-A3B, has 30,532,122,624 parameters. Only 3.35B (11.0%) run for each token: attention, the router, 8 of 128 experts per layer and the embeddings. The idle experts are 89.0% of the model. A forward pass costs 6.1 GFLOP per token, against 60 if every expert ran, but all 56.87 GiB of bf16 weights must stay in memory. With 4 KV heads of 128 its cache is 96 KiB per token, 3.00 GiB at 32K and 12.00 GiB at 128K. Qwen3-235B-A22B runs 22.19B of 235.09B (9.4%) and needs 437.90 GiB of weights but only 188 KiB of cache per token. Compare the attention layouts at 128K: Qwen-7B with full MHA needs 512 KiB per token (64.00 GiB), Qwen2.5-7B with 4 KV heads 56 KiB (7.00 GiB). The small Qwen3-0.6B needs 112 KiB, twice Qwen2.5-7B, because it keeps 8 KV heads of 128. Its QK-Norm gains are 7,168 parameters, Qwen2.5-7B's QKV biases 129,024. Qwen3-8B saw about 4,395 training tokens per parameter.

## Worked example: one layer of Qwen3-30B-A3B

The config has $d=2048$, 48 layers, 32 query and 4 KV heads with $d_h=128$, 128 experts of width 768 with 8 active, a 151,936-row vocabulary, and an untied head. Note that $32\cdot128=4096\ne d$: Qwen3 sets the head size independently, so $W_q$ maps 2048 to 4096.

Parameters per layer (computed in Python; the totals match the model card's 30.5B and 3.3B)

| part | formula | stored | used per token |
| --- | --- | --- | --- |
| W_q, W_o | 2 · 2048 · 4096 | 16,777,216 | 16,777,216 |
| W_k, W_v | 2 · 2048 · 512 | 4,194,304 | 4,194,304 |
| QK-Norm gains | 2 · 128 | 256 | 256 |
| router | 2048 · 128 | 262,144 | 262,144 |
| experts | 128 (or 8) · 3 · 2048 · 768 | 603,979,776 | 37,748,736 |
| 2 RMSNorm gains | 2 · 2048 | 4,096 | 4,096 |
| one layer |  | 623,120,640 | 56,889,600 |
| × 48 + embeddings 622,329,856 + final norm |  | 30,532,122,624 | 3,353,032,704 |

**From parameters to serving cost**

1. Experts are 96.9% of a layer's stored weights but only 8/128 of them run, so each layer stores 623.1M parameters and uses 56.9M. Over the model, 11.0% of the weights do the work for any one token.
2. Compute per generated token: about $2\times$ the active matmul weights, $2\cdot(48\cdot56.9\text{M}+151{,}936\cdot2048)\approx6.1$ GFLOP, which is close to what a dense 3B model costs.
3. Memory: all 30.5B parameters must be resident, $30.53\times10^9\cdot2$ bytes $=56.87$ GiB in bf16, close to what a dense 30B model needs.
4. KV cache: $2\cdot48\cdot4\cdot128\cdot2=98{,}304$ bytes = 96 KiB per token, so a 32K context costs 3.00 GiB and a 128K context (YaRN $\times4$) 12.00 GiB.
5. YaRN at $s=4$ with base $10^6$, $d_h=128$, $L_{orig}=32768$: by the paper's ramp ($\alpha=1$, $\beta=32$), 24 of the 64 frequency pairs make more than 32 turns in 32K and are kept, 24 make less than one and are divided by 4, and 16 are blended. The logits are multiplied by $1.296$. Implementations round the ramp boundaries, so the split can differ by a pair.

## Qwen3: thinking and non-thinking in one model

Qwen3's post-trained models can do both. In *thinking mode* they write a reasoning trace inside `<think> … </think>` before the answer. In *non-thinking mode* they answer directly, and the chat template inserts an empty think block. The report describes four post-training stages for the flagship models: a long chain-of-thought cold start, reasoning RL, *thinking-mode fusion* (SFT on a mix of thinking and non-thinking data, so that one model learns both formats), and general RL. The smaller models are trained by strong-to-weak distillation from the large ones instead of running the full pipeline. A user switches modes with `enable_thinking` in the chat template, or per turn with `/think` and `/no_think`.

Because the two modes share weights, the report also describes a **thinking budget**. When the trace reaches a user-set number of tokens, generation is interrupted, a short instruction to stop thinking is inserted, and the model writes its answer from the partial trace. Accuracy rises smoothly with the budget, so a deployment can trade latency for quality per request. In July 2025 the Qwen team released updated "2507" checkpoints that split the instruct and thinking variants into separate models again.

```python
from transformers import AutoModelForCausalLM, AutoTokenizer

name = "Qwen/Qwen3-8B"
tok = AutoTokenizer.from_pretrained(name)
model = AutoModelForCausalLM.from_pretrained(name, torch_dtype="auto", device_map="auto")

def chat(prompt, thinking=True, budget=4096):
    text = tok.apply_chat_template(
        [{"role": "user", "content": prompt}],
        tokenize=False, add_generation_prompt=True,
        enable_thinking=thinking,          # False: the template adds an empty <think></think>
    )
    ids = tok(text, return_tensors="pt").to(model.device)
    # model card: thinking T=0.6, top-p 0.95, top-k 20; non-thinking T=0.7, top-p 0.8, top-k 20
    out = model.generate(**ids, max_new_tokens=budget, do_sample=True,
                         temperature=0.6 if thinking else 0.7,
                         top_p=0.95 if thinking else 0.8, top_k=20)
    new = out[0][ids.input_ids.shape[1]:].tolist()
    try:                                   # split at the last </think> (token id 151668)
        cut = len(new) - new[::-1].index(151668)
    except ValueError:
        cut = 0
    return tok.decode(new[:cut], skip_special_tokens=True).strip(), \
           tok.decode(new[cut:], skip_special_tokens=True).strip()

trace, answer = chat("How many positive divisors does 360 have?")
_, quick = chat("Translate 'good morning' into French.", thinking=False)
_, also_quick = chat("Is 2^31 - 1 prime? /no_think")   # per-turn soft switch

# Beyond the native 32K: static YaRN x4 (the scaling applies to every input,
# so the model card advises enabling it only when long inputs are expected)
long_model = AutoModelForCausalLM.from_pretrained(
    name, torch_dtype="auto", device_map="auto",
    rope_scaling={"rope_type": "yarn", "factor": 4.0, "original_max_position_embeddings": 32768},
    max_position_embeddings=131072,
)

# Parameter count from the config (8.19B for Qwen3-8B, 6.95B without embeddings)
print(sum(p.numel() for p in model.parameters()) / 1e9)
```

Qwen3 with Hugging Face transformers: thinking vs non-thinking, separating the trace, and the YaRN setting for 128K (follows the model card; not run here)

> **Static YaRN hurts short inputs:** Hugging Face applies `rope_scaling` to every sequence, not only long ones. With factor 4, every short prompt is also run through rescaled frequencies and the logit temperature. The Qwen model cards warn that this "may impact performance on shorter texts" and recommend enabling it only when long contexts are needed, with a factor matched to the length, e.g. 2.0 for 64K.

> **Same tokenizer, more data:** From Qwen2 to Qwen2.5, the 7B and 72B architectures are identical: same layers, widths, heads and vocabulary, so the parameter count is the same 7.61B. The jump in quality came from 18T instead of 7T pre-training tokens and better post-training. Qwen3-8B saw about 4,400 tokens per parameter, over 200 times Chinchilla's 20.

> **Do not decode greedily in thinking mode:** The Qwen3 model cards advise against greedy decoding with thinking enabled, because it can cause endless repetition. They recommend temperature 0.6, top-p 0.95 and top-k 20. Long traces also consume the context: a 32K budget of thinking tokens costs the same KV cache as a 32K prompt.

**Strengths**

- ~152K byte-level BPE vocabulary: efficient on Chinese and many other languages, no unknown tokens
- Full size ladder from 0.5B/0.6B to 72B dense and 235B MoE, many under Apache 2.0
- GQA and a RoPE base of 10⁶ plus YaRN give 128K context at a modest KV cache (56 KiB/token for Qwen2.5-7B)
- MoE models run 3B or 22B active parameters at the quality of much larger dense models
- Qwen3: one set of weights for both fast answers and long reasoning, with a controllable thinking budget

**Limitations**

- Large vocabulary dominates small models (27.6% of Qwen2.5-0.5B even when tied)
- QKV bias (Qwen–Qwen2.5) and QK-Norm (Qwen3) mean a plain Llama implementation is not weight-compatible
- MoE saves compute, not memory: Qwen3-235B-A22B needs about 438 GiB of bf16 weights
- Static YaRN in common runtimes degrades short-context quality unless switched on only when needed
- Thinking mode multiplies output length, latency and KV-cache use

**Question:** Qwen3-30B-A3B has 30.5B parameters but '3B active'. What does serving it cost, compared with a dense model?

- A. Compute and memory of a 3B model
- B. Compute of roughly a 3B model per token, but memory for all 30.5B weights
- C. Compute of a 30B model, memory of a 3B model
- D. Compute and memory of a 30B model, but higher quality

**Answer:** B. Compute of roughly a 3B model per token, but memory for all 30.5B weights — Each token runs only the router's top-8 of 128 experts per layer, about 3.35B parameters or 6.1 GFLOP. But any token may pick any expert, so all 30.5B parameters (56.87 GiB in bf16) must be resident. MoE trades memory for compute.

**Question:** What did Qwen3 change in the attention projections relative to Qwen2.5?

- A. It added biases to q, k and v
- B. It removed the q/k/v biases and added RMSNorm on each head's queries and keys (QK-Norm)
- C. It replaced RoPE with ALiBi
- D. It switched from GQA back to MHA

**Answer:** B. It removed the q/k/v biases and added RMSNorm on each head's queries and keys (QK-Norm) — Qwen through Qwen2.5 kept biases on the q, k, v projections, which the original report linked to better length extrapolation. Qwen3 drops them and normalises $q$ and $k$ per head before RoPE. This bounds the attention logits and stabilises training.

**Key points**

- Qwen is a Llama-style decoder (pre-RMSNorm, SwiGLU, RoPE, untied head) with one ~152K byte-level BPE vocabulary (151,643 regular tokens) across all generations.
- Attention: QKV bias in Qwen–Qwen2.5, replaced by per-head QK-Norm in Qwen3; MHA in Qwen, GQA in Qwen1.5's largest models and at every size from Qwen2.
- Long context: RoPE base $10^4\to10^6$ during 32K pre-training, then YaRN factor 4 (logits $\times1.296$) and Dual Chunk Attention to reach about 128K.
- Data grew from 2–3T (Qwen) to 7T (Qwen2), 18T (Qwen2.5) and 36T tokens in 119 languages (Qwen3); MoE variants range from Qwen1.5-MoE-A2.7B to Qwen3-235B-A22B (128 experts, 8 active).
- Qwen3 fuses thinking and non-thinking modes in one model, switched by enable_thinking or /think and /no_think, with a thinking budget.

---

From MLFoundry (https://www.mlfoundry.in). The web page adds interactive figures and diagrams.
