Qwen

Qwen (Alibaba, 2023–2025) is a Llama-style decoder family with one ~152K byte-level BPE vocabulary, QKV bias until Qwen3 replaced it with QK-Norm, GQA from Qwen2, RoPE base 10⁴→10⁶ plus YaRN and DCA for 128K context, MoE variants, and 2.4T→36T training tokens.

15 min read1363 wordsadvanced0 viewsupdated 2026-10-13

Definition

Qwen (通义千问, Tongyi Qianwen) is the open-weight language-model series of Alibaba Cloud's Qwen team. It is documented in Bai et al., "Qwen Technical Report" (2023), Yang et al., "Qwen2 Technical Report" (2024), the Qwen team's "Qwen2.5 Technical Report" (2024), and Yang et al., "Qwen3 Technical Report" (2025). Qwen1.5 (February 2024) was released with a blog post rather than a paper. Every generation is a Decoder-only transformer in the Llama mould: pre-RMSNorm, SwiGLU, RoPE and an untied head. All share one byte-level BPE tokenizer of about 152K tokens. The generations differ in a few block details (QKV bias, then QK-Norm), attention layout (MHA, then GQA), long-context method, mixture-of-experts variants and, above all, the amount of data: about 2–3T tokens for Qwen, 7T for Qwen2, 18T for Qwen2.5 and 36T for Qwen3.

Qwen began in 2023 as Alibaba's answer to LLaMA. It aimed at strong English and Chinese, plus code and maths, with open weights. LLaMA's 32K SentencePiece vocabulary covers Chinese poorly: a sentence breaks into many byte-fallback pieces, which wastes context and compute. Qwen therefore started from a large BPE vocabulary and kept it fixed for later generations, while scaling everything else. Later releases grew to 119 languages and dialects (Qwen3), contexts of 128K tokens and more, a 235B mixture-of-experts model, and a single model that can either reason step by step or answer directly.

The block and what changed

Qwen3 blockoutput head (tied in small models)RMSNorm (final)× Ldense SwiGLU, or MoE:128 experts, top-8, no sharedRMSNormcausal attention, then W_o (no bias)GQA at every sizeRoPE, base 1,000,000QK-Norm: RMSNorm over each headof q and k (d_h = 128 gains each)W_q, W_k, W_vno biasRMSNorm++token embeddingbyte-level BPE, 151,643 regular tokenssame tokenizer in every generation; orange outline =changed from the previous onefive generations (hover or click)Qwen · 20231.8B 7B 14B 72B · up to 3T tokensMHA · QKV bias · NTK + LogN at inferenceQwen1.5 · Feb 20240.5B–110B + MoE-A2.7B · 32K contextGQA only in 32B and 110B · QKV biasQwen2 · Jun 20240.5B 1.5B 7B 72B + 57B-A14B · 7TGQA everywhere · YaRN + DCA to 128KQwen2.5 · Sep 20240.5B–72B dense · 18T tokens32K native, 128K via YaRN ×4Qwen3 · Apr 20250.6B–32B + 30B-A3B, 235B-A22B · 36TQK-Norm, no QKV bias · thinking modesevery generation: pre-RMSNorm, SwiGLU,RoPE, no MLP biases, as in Llama
fig.The Qwen block, redrawn for each generation; hover or click a generation on the right, and the parts that changed from the previous one get an orange outline. Qwen (2023) is LLaMA's block with full multi-head attention, biases on the q, k and v projections, and RoPE with base 10,000. Qwen1.5 adds GQA only in the 32B and 110B models and a first MoE model (60 routed experts, top-4, plus a shared expert). Qwen2 uses GQA at every size, raises the RoPE base to 1,000,000 and adds a 64-expert MoE. Qwen2.5 keeps the architecture and scales the data to 18T tokens. Qwen3 drops the QKV bias in favour of QK-Norm and uses 128-expert MoE layers with no shared expert. The tokenizer is the same throughout.
Qwen–Qwen2.5:  q=Wqx+bq, k=Wkx+bk, v=Wvx+bvQwen3:  q=RMSNormq(Wqx), k=RMSNormk(Wkx), v=Wvx\textbf{Qwen–Qwen2.5: }\ q=W_qx+b_q,\ k=W_kx+b_k,\ v=W_vx+b_v\qquad\quad \textbf{Qwen3: }\ q=\mathrm{RMSNorm}_q(W_qx),\ k=\mathrm{RMSNorm}_k(W_kx),\ v=W_vx
bq,bk,bvb_q,b_k,b_v
the only biases in the model. The Qwen report adds them "to enhance the extrapolation ability" of RoPE; for Qwen2.5-7B they are 129,024 parameters, 0.0017% of the model
RMSNormq, RMSNormk\mathrm{RMSNorm}_q,\ \mathrm{RMSNorm}_k
QK-Norm: RMSNorm over each head's dh=128d_h=128 coordinates, applied before RoPE, with one learned gain vector shared by all heads
xx
the pre-RMSNorm-ed residual stream
QK-Norm\text{QK-Norm}
bounds the size of q⊤kq^\top k, so attention logits cannot blow up in bf16. Qwen3 adopts it "to ensure stable training" (the idea is from Dehghani et al.'s ViT-22B, 2023)

The QKV bias is a small detail with a practical consequence. Any code that re-implements the block from a plain Llama definition will silently drop bq,bk,bvb_q,b_k,b_v and get wrong outputs. This is why Hugging Face has separate Qwen2 and Qwen3 model classes, although both are otherwise Llama-shaped.

The tokenizer

Qwen's tokenizer starts from the cl100k_base byte-level BPE of OpenAI's tiktoken (see Byte-pair encoding (BPE)) and adds frequently used Chinese characters and words, plus tokens for other languages. Numbers are split into single digits. Byte-level means any UTF-8 string is representable, so there is never an unknown token. The Qwen2 report gives the vocabulary as 151,643 regular tokens plus 3 control tokens. Qwen2.5 expands the control tokens to 22, and Qwen3 adds <think>-style tokens for reasoning. Checkpoints pad the embedding to 151,936 or 152,064 rows. The Qwen report showed that this vocabulary compresses Chinese, and most of the other languages it tested, better than LLaMA's 32K tokenizer, which means more text per context window and per FLOP.

embedding share=V d (2−1tied)N:Qwen2.5-0.5B: 151936⋅896494,032,768=27.6%,Qwen2.5-7B: 2⋅152064⋅35847,615,616,512=14.3%\text{embedding share}=\frac{V\,d\,(2-\mathbb 1_{\text{tied}})}{N}:\qquad \text{Qwen2.5-0.5B: }\frac{151936\cdot896}{494{,}032{,}768}=27.6\%,\qquad \text{Qwen2.5-7B: }\frac{2\cdot152064\cdot3584}{7{,}615{,}616{,}512}=14.3\%
VV
embedding rows in the checkpoint (padded vocabulary)
1tied\mathbb 1_{\text{tied}}
1 when the head reuses the embedding; Qwen ties it only in the smallest models (Qwen2.5 0.5B–3B, Qwen3 0.6B–4B)
NN
total parameters

A 152K vocabulary is expensive for small models. Untied, Qwen2.5-0.5B would spend 43% of its parameters on the two tables. Tying, and calling the rest "non-embedding parameters" (0.36B), is how the model cards present it.

Long context: RoPE base, YaRN and dual chunk attention

Qwen (2023) trained at 2048 tokens and extended at inference with training-free tricks. It used dynamic NTK-aware interpolation (see Rotary positional embeddings (RoPE)), LogN attention scaling, which multiplies the logits by log⁡Ltrainn\log_{L_{train}}n when the length nn exceeds the training length, and layer-wise window attention. From Qwen2 on, the recipe has two steps. First, the final pre-training stage raises the context to 32K and the RoPE base from 10,000 to 1,000,000 (adjusted base frequency, ABF), so that low-frequency pairs rotate slowly enough to keep long distances apart. Second, inference uses YaRN with factor 4 to reach about 128K, together with Dual Chunk Attention (DCA; An et al., "Training-Free Long-Context Scaling of Large Language Models", 2024). DCA splits a long sequence into chunks and remaps relative positions so that no query–key distance exceeds what the model saw in training.

θi=b−2i/dh,θi′=(1−γ(ri))θis+γ(ri) θi,ri=Lorig θi2π,logits×(0.1ln⁡s+1)2\theta_i=b^{-2i/d_h},\qquad \theta'_i=\big(1-\gamma(r_i)\big)\frac{\theta_i}{s}+\gamma(r_i)\,\theta_i,\qquad r_i=\frac{L_{orig}\,\theta_i}{2\pi},\qquad \text{logits}\times\big(0.1\ln s+1\big)^2
bb
RoPE base: 10410^4 in Qwen, 10610^6 from Qwen2 (raised during long-context pre-training)
s=4, Lorig=32768s=4,\ L_{orig}=32768
Qwen2.5 and Qwen3 model cards: rope_scaling = {"type": "yarn", "factor": 4.0, "original_max_position_embeddings": 32768}
γ(r)\gamma(r)
YaRN's ramp: 0 for pairs making fewer than 1 turn within LorigL_{orig} (fully interpolated), 1 above 32 turns (kept), linear between
(0.1ln⁡4+1)2(0.1\ln4+1)^2
=1.296=1.296, YaRN's attention temperature at s=4s=4

Qwen2.5-Turbo and the open Qwen2.5-7B/14B-Instruct-1M models (early 2025) pushed this to 1M tokens. They used progressive long-context training plus DCA and sparse attention at inference.

Generations

Qwen (2023)
1.8B, 7B, 14B, 72B. 7B: L=32L=32, d=4096d=4096, 32 heads (MHA), dff=11008d_{ff}=11008; 14B: 40 layers, d=5120d=5120, 40 heads. Up to 3T tokens (2.4T for 7B), context 2048 in pre-training. Chat models via SFT and RLHF, and code and maths specialists (Code-Qwen, Math-Qwen)
Qwen1.5 (Feb 2024)
0.5B to 110B dense, 32K context for all sizes; GQA only in 32B and 110B. Qwen1.5-MoE-A2.7B: 14.3B total, 2.7B active, 60 routed experts (top-4) plus a shared expert as wide as four of them
Qwen2 (Jun 2024)
0.5B, 1.5B, 7B, 72B and 57B-A14B (64 experts, top-8, plus shared). GQA at every size; 7T tokens (12T for 0.5B); about 30 languages. 72B: L=80L=80, d=8192d=8192, 64 query / 8 KV heads, dff=29568d_{ff}=29568
Qwen2.5 (Sep 2024)
0.5B, 1.5B, 3B, 7B, 14B, 32B, 72B; 18T tokens. Same architecture as Qwen2 (Qwen2.5-7B: 28 layers, d=3584d=3584, 28/4 heads, 7.61B parameters). 128K context with YaRN and up to 8K generated tokens; Qwen2.5-Turbo and -Plus are API-only MoE models
Qwen3 (Apr 2025)
dense 0.6B, 1.7B, 4B, 8B, 14B, 32B; MoE 30B-A3B and 235B-A22B (128 experts, 8 active, no shared expert, global-batch load-balancing loss). 36T tokens in 119 languages and dialects; QK-Norm, no QKV bias. Apache 2.0
Qwen3 pre-training
three stages: general data at 4K sequence length (over 30T tokens); a knowledge- and reasoning-heavy stage of about 5T tokens; then long-context training at 32K with the RoPE base raised from 10,000 to 1,000,000
Qwen3-235B-A22B
L=94L=94, d=4096d=4096, 64 query / 4 KV heads of 128, experts of width 1536; 235B total, 22B active per token

Mixture of experts in Qwen

Each Qwen MoE layer replaces the dense SwiGLU with EE small SwiGLU experts and a linear router (see Mixture of experts (MoE) and Sparse MoE routing). The router picks the top-kk experts per token and mixes their outputs with softmax weights. Qwen1.5-MoE and Qwen2-MoE also have an always-on shared expert whose output passes through a sigmoid gate. Its job is to hold common knowledge so that the routed experts can specialise. Qwen3 drops the shared expert and balances the experts with a load-balancing loss computed over the whole global batch rather than per micro-batch, which allows experts to specialise by domain.

Qwen3-30B-A3B: L = 48 · d = 2048 · 32 q / 4 kv heads of 128 · V = 151,936MoE: 128 routed experts, top-8, width 768, no shared expert · QK-Norm, no bias · untied headtotal 30,532,122,624 (30.53B) · active per token 3.35B (11.0%)■ embedding + head 2.0%■ attention 3.0%■ router + shared 0.041%■ active experts 5.9%■ idle experts 89.0%non-embedding parameters29.91Bq/k RMSNorm gains12,288 (0.000040% of the model)bf16 weights to hold56.87 GiBforward FLOPs per token6.1 GFLOP (all experts: 60)KV cache per token (bf16)96 KiBKV cache at 32K tokens3.00 GiBpre-training tokens36T · 1,179 per parametercontext32K, 128K with YaRN ×4active vs total parameterslog-log; dense models on the diagonal1B1B10B10B100B100Btotal →↑ active● dense ● MoE · click a point8 of 128 experts run per token: compute follows active, memory follows total
model
fig.Exact parameter counts for eleven released checkpoints, computed from their configs. The totals match the model cards: Qwen2.5-7B 7.61B (6.53B non-embedding), Qwen3-8B 8.19B, Qwen3-30B-A3B 30.5B with 3.3B active, and Qwen3-235B-A22B 235B with 22B active. The default, Qwen3-30B-A3B, has 30,532,122,624 parameters. Only 3.35B (11.0%) run for each token: attention, the router, 8 of 128 experts per layer and the embeddings. The idle experts are 89.0% of the model. A forward pass costs 6.1 GFLOP per token, against 60 if every expert ran, but all 56.87 GiB of bf16 weights must stay in memory. With 4 KV heads of 128 its cache is 96 KiB per token, 3.00 GiB at 32K and 12.00 GiB at 128K. Qwen3-235B-A22B runs 22.19B of 235.09B (9.4%) and needs 437.90 GiB of weights but only 188 KiB of cache per token. Compare the attention layouts at 128K: Qwen-7B with full MHA needs 512 KiB per token (64.00 GiB), Qwen2.5-7B with 4 KV heads 56 KiB (7.00 GiB). The small Qwen3-0.6B needs 112 KiB, twice Qwen2.5-7B, because it keeps 8 KV heads of 128. Its QK-Norm gains are 7,168 parameters, Qwen2.5-7B's QKV biases 129,024. Qwen3-8B saw about 4,395 training tokens per parameter.

Worked example: one layer of Qwen3-30B-A3B

The config has d=2048d=2048, 48 layers, 32 query and 4 KV heads with dh=128d_h=128, 128 experts of width 768 with 8 active, a 151,936-row vocabulary, and an untied head. Note that 32⋅128=4096≠d32\cdot128=4096\ne d: Qwen3 sets the head size independently, so WqW_q maps 2048 to 4096.

partformulastoredused per token
W_q, W_o2 · 2048 · 409616,777,21616,777,216
W_k, W_v2 · 2048 · 5124,194,3044,194,304
QK-Norm gains2 · 128256256
router2048 · 128262,144262,144
experts128 (or 8) · 3 · 2048 · 768603,979,77637,748,736
2 RMSNorm gains2 · 20484,0964,096
one layer623,120,64056,889,600
× 48 + embeddings 622,329,856 + final norm30,532,122,6243,353,032,704
Parameters per layer (computed in Python; the totals match the model card's 30.5B and 3.3B)

From parameters to serving cost

  1. 1Experts are 96.9% of a layer's stored weights but only 8/128 of them run, so each layer stores 623.1M parameters and uses 56.9M. Over the model, 11.0% of the weights do the work for any one token.
  2. 2Compute per generated token: about 2×2\times the active matmul weights, 2⋅(48⋅56.9M+151,936⋅2048)≈6.12\cdot(48\cdot56.9\text{M}+151{,}936\cdot2048)\approx6.1 GFLOP, which is close to what a dense 3B model costs.
  3. 3Memory: all 30.5B parameters must be resident, 30.53×109⋅230.53\times10^9\cdot2 bytes =56.87=56.87 GiB in bf16, close to what a dense 30B model needs.
  4. 4KV cache: 2⋅48⋅4⋅128⋅2=98,3042\cdot48\cdot4\cdot128\cdot2=98{,}304 bytes = 96 KiB per token, so a 32K context costs 3.00 GiB and a 128K context (YaRN ×4\times4) 12.00 GiB.
  5. 5YaRN at s=4s=4 with base 10610^6, dh=128d_h=128, Lorig=32768L_{orig}=32768: by the paper's ramp (α=1\alpha=1, β=32\beta=32), 24 of the 64 frequency pairs make more than 32 turns in 32K and are kept, 24 make less than one and are divided by 4, and 16 are blended. The logits are multiplied by 1.2961.296. Implementations round the ramp boundaries, so the split can differ by a pair.

Qwen3: thinking and non-thinking in one model

Qwen3's post-trained models can do both. In thinking mode they write a reasoning trace inside <think> … </think> before the answer. In non-thinking mode they answer directly, and the chat template inserts an empty think block. The report describes four post-training stages for the flagship models: a long chain-of-thought cold start, reasoning RL, thinking-mode fusion (SFT on a mix of thinking and non-thinking data, so that one model learns both formats), and general RL. The smaller models are trained by strong-to-weak distillation from the large ones instead of running the full pipeline. A user switches modes with enable_thinking in the chat template, or per turn with /think and /no_think.

Because the two modes share weights, the report also describes a thinking budget. When the trace reaches a user-set number of tokens, generation is interrupted, a short instruction to stop thinking is inserted, and the model writes its answer from the partial trace. Accuracy rises smoothly with the budget, so a deployment can trade latency for quality per request. In July 2025 the Qwen team released updated "2507" checkpoints that split the instruct and thinking variants into separate models again.

python
1from transformers import AutoModelForCausalLM, AutoTokenizer
2
3name = "Qwen/Qwen3-8B"
4tok = AutoTokenizer.from_pretrained(name)
5model = AutoModelForCausalLM.from_pretrained(name, torch_dtype="auto", device_map="auto")
6
7def chat(prompt, thinking=True, budget=4096):
8 text = tok.apply_chat_template(
9 [{"role": "user", "content": prompt}],
10 tokenize=False, add_generation_prompt=True,
11 enable_thinking=thinking, # False: the template adds an empty <think></think>
12 )
13 ids = tok(text, return_tensors="pt").to(model.device)
14 # model card: thinking T=0.6, top-p 0.95, top-k 20; non-thinking T=0.7, top-p 0.8, top-k 20
15 out = model.generate(**ids, max_new_tokens=budget, do_sample=True,
16 temperature=0.6 if thinking else 0.7,
17 top_p=0.95 if thinking else 0.8, top_k=20)
18 new = out[0][ids.input_ids.shape[1]:].tolist()
19 try: # split at the last </think> (token id 151668)
20 cut = len(new) - new[::-1].index(151668)
21 except ValueError:
22 cut = 0
23 return tok.decode(new[:cut], skip_special_tokens=True).strip(), \
24 tok.decode(new[cut:], skip_special_tokens=True).strip()
25
26trace, answer = chat("How many positive divisors does 360 have?")
27_, quick = chat("Translate 'good morning' into French.", thinking=False)
28_, also_quick = chat("Is 2^31 - 1 prime? /no_think") # per-turn soft switch
29
30# Beyond the native 32K: static YaRN x4 (the scaling applies to every input,
31# so the model card advises enabling it only when long inputs are expected)
32long_model = AutoModelForCausalLM.from_pretrained(
33 name, torch_dtype="auto", device_map="auto",
34 rope_scaling={"rope_type": "yarn", "factor": 4.0, "original_max_position_embeddings": 32768},
35 max_position_embeddings=131072,
36)
37
38# Parameter count from the config (8.19B for Qwen3-8B, 6.95B without embeddings)
39print(sum(p.numel() for p in model.parameters()) / 1e9)
Qwen3 with Hugging Face transformers: thinking vs non-thinking, separating the trace, and the YaRN setting for 128K (follows the model card; not run here)

Use when

  • +~152K byte-level BPE vocabulary: efficient on Chinese and many other languages, no unknown tokens
  • +Full size ladder from 0.5B/0.6B to 72B dense and 235B MoE, many under Apache 2.0
  • +GQA and a RoPE base of 10⁶ plus YaRN give 128K context at a modest KV cache (56 KiB/token for Qwen2.5-7B)
  • +MoE models run 3B or 22B active parameters at the quality of much larger dense models
  • +Qwen3: one set of weights for both fast answers and long reasoning, with a controllable thinking budget

Avoid when

  • −Large vocabulary dominates small models (27.6% of Qwen2.5-0.5B even when tied)
  • −QKV bias (Qwen–Qwen2.5) and QK-Norm (Qwen3) mean a plain Llama implementation is not weight-compatible
  • −MoE saves compute, not memory: Qwen3-235B-A22B needs about 438 GiB of bf16 weights
  • −Static YaRN in common runtimes degrades short-context quality unless switched on only when needed
  • −Thinking mode multiplies output length, latency and KV-cache use

Check yourself

Qwen3-30B-A3B has 30.5B parameters but '3B active'. What does serving it cost, compared with a dense model?

Check yourself

What did Qwen3 change in the attention projections relative to Qwen2.5?

Remember

  • ✓Qwen is a Llama-style decoder (pre-RMSNorm, SwiGLU, RoPE, untied head) with one ~152K byte-level BPE vocabulary (151,643 regular tokens) across all generations.
  • ✓Attention: QKV bias in Qwen–Qwen2.5, replaced by per-head QK-Norm in Qwen3; MHA in Qwen, GQA in Qwen1.5's largest models and at every size from Qwen2.
  • ✓Long context: RoPE base 104→10610^4\to10^6 during 32K pre-training, then YaRN factor 4 (logits ×1.296\times1.296) and Dual Chunk Attention to reach about 128K.
  • ✓Data grew from 2–3T (Qwen) to 7T (Qwen2), 18T (Qwen2.5) and 36T tokens in 119 languages (Qwen3); MoE variants range from Qwen1.5-MoE-A2.7B to Qwen3-235B-A22B (128 experts, 8 active).
  • ✓Qwen3 fuses thinking and non-thinking modes in one model, switched by enable_thinking or /think and /no_think, with a thinking budget.
← back to Deep Learning roadmap