Definition
Qwen (通义千问, Tongyi Qianwen) is the open-weight language-model series of Alibaba Cloud's Qwen team. It is documented in Bai et al., "Qwen Technical Report" (2023), Yang et al., "Qwen2 Technical Report" (2024), the Qwen team's "Qwen2.5 Technical Report" (2024), and Yang et al., "Qwen3 Technical Report" (2025). Qwen1.5 (February 2024) was released with a blog post rather than a paper. Every generation is a Decoder-only transformer in the Llama mould: pre-RMSNorm, SwiGLU, RoPE and an untied head. All share one byte-level BPE tokenizer of about 152K tokens. The generations differ in a few block details (QKV bias, then QK-Norm), attention layout (MHA, then GQA), long-context method, mixture-of-experts variants and, above all, the amount of data: about 2–3T tokens for Qwen, 7T for Qwen2, 18T for Qwen2.5 and 36T for Qwen3.
Qwen began in 2023 as Alibaba's answer to LLaMA. It aimed at strong English and Chinese, plus code and maths, with open weights. LLaMA's 32K SentencePiece vocabulary covers Chinese poorly: a sentence breaks into many byte-fallback pieces, which wastes context and compute. Qwen therefore started from a large BPE vocabulary and kept it fixed for later generations, while scaling everything else. Later releases grew to 119 languages and dialects (Qwen3), contexts of 128K tokens and more, a 235B mixture-of-experts model, and a single model that can either reason step by step or answer directly.
The block and what changed
- the only biases in the model. The Qwen report adds them "to enhance the extrapolation ability" of RoPE; for Qwen2.5-7B they are 129,024 parameters, 0.0017% of the model
- QK-Norm: RMSNorm over each head's coordinates, applied before RoPE, with one learned gain vector shared by all heads
- the pre-RMSNorm-ed residual stream
- bounds the size of , so attention logits cannot blow up in bf16. Qwen3 adopts it "to ensure stable training" (the idea is from Dehghani et al.'s ViT-22B, 2023)
The QKV bias is a small detail with a practical consequence. Any code that re-implements the block from a plain Llama definition will silently drop and get wrong outputs. This is why Hugging Face has separate Qwen2 and Qwen3 model classes, although both are otherwise Llama-shaped.
The tokenizer
Qwen's tokenizer starts from the cl100k_base byte-level BPE of OpenAI's tiktoken (see Byte-pair encoding (BPE)) and adds frequently used Chinese characters and words, plus tokens for other languages. Numbers are split into single digits. Byte-level means any UTF-8 string is representable, so there is never an unknown token. The Qwen2 report gives the vocabulary as 151,643 regular tokens plus 3 control tokens. Qwen2.5 expands the control tokens to 22, and Qwen3 adds <think>-style tokens for reasoning. Checkpoints pad the embedding to 151,936 or 152,064 rows. The Qwen report showed that this vocabulary compresses Chinese, and most of the other languages it tested, better than LLaMA's 32K tokenizer, which means more text per context window and per FLOP.
- embedding rows in the checkpoint (padded vocabulary)
- 1 when the head reuses the embedding; Qwen ties it only in the smallest models (Qwen2.5 0.5B–3B, Qwen3 0.6B–4B)
- total parameters
A 152K vocabulary is expensive for small models. Untied, Qwen2.5-0.5B would spend 43% of its parameters on the two tables. Tying, and calling the rest "non-embedding parameters" (0.36B), is how the model cards present it.
Long context: RoPE base, YaRN and dual chunk attention
Qwen (2023) trained at 2048 tokens and extended at inference with training-free tricks. It used dynamic NTK-aware interpolation (see Rotary positional embeddings (RoPE)), LogN attention scaling, which multiplies the logits by when the length exceeds the training length, and layer-wise window attention. From Qwen2 on, the recipe has two steps. First, the final pre-training stage raises the context to 32K and the RoPE base from 10,000 to 1,000,000 (adjusted base frequency, ABF), so that low-frequency pairs rotate slowly enough to keep long distances apart. Second, inference uses YaRN with factor 4 to reach about 128K, together with Dual Chunk Attention (DCA; An et al., "Training-Free Long-Context Scaling of Large Language Models", 2024). DCA splits a long sequence into chunks and remaps relative positions so that no query–key distance exceeds what the model saw in training.
- RoPE base: in Qwen, from Qwen2 (raised during long-context pre-training)
- Qwen2.5 and Qwen3 model cards:
rope_scaling = {"type": "yarn", "factor": 4.0, "original_max_position_embeddings": 32768} - YaRN's ramp: 0 for pairs making fewer than 1 turn within (fully interpolated), 1 above 32 turns (kept), linear between
- , YaRN's attention temperature at
Qwen2.5-Turbo and the open Qwen2.5-7B/14B-Instruct-1M models (early 2025) pushed this to 1M tokens. They used progressive long-context training plus DCA and sparse attention at inference.
Generations
- Qwen (2023)
- 1.8B, 7B, 14B, 72B. 7B: , , 32 heads (MHA), ; 14B: 40 layers, , 40 heads. Up to 3T tokens (2.4T for 7B), context 2048 in pre-training. Chat models via SFT and RLHF, and code and maths specialists (Code-Qwen, Math-Qwen)
- Qwen1.5 (Feb 2024)
- 0.5B to 110B dense, 32K context for all sizes; GQA only in 32B and 110B. Qwen1.5-MoE-A2.7B: 14.3B total, 2.7B active, 60 routed experts (top-4) plus a shared expert as wide as four of them
- Qwen2 (Jun 2024)
- 0.5B, 1.5B, 7B, 72B and 57B-A14B (64 experts, top-8, plus shared). GQA at every size; 7T tokens (12T for 0.5B); about 30 languages. 72B: , , 64 query / 8 KV heads,
- Qwen2.5 (Sep 2024)
- 0.5B, 1.5B, 3B, 7B, 14B, 32B, 72B; 18T tokens. Same architecture as Qwen2 (Qwen2.5-7B: 28 layers, , 28/4 heads, 7.61B parameters). 128K context with YaRN and up to 8K generated tokens; Qwen2.5-Turbo and -Plus are API-only MoE models
- Qwen3 (Apr 2025)
- dense 0.6B, 1.7B, 4B, 8B, 14B, 32B; MoE 30B-A3B and 235B-A22B (128 experts, 8 active, no shared expert, global-batch load-balancing loss). 36T tokens in 119 languages and dialects; QK-Norm, no QKV bias. Apache 2.0
- Qwen3 pre-training
- three stages: general data at 4K sequence length (over 30T tokens); a knowledge- and reasoning-heavy stage of about 5T tokens; then long-context training at 32K with the RoPE base raised from 10,000 to 1,000,000
- Qwen3-235B-A22B
- , , 64 query / 4 KV heads of 128, experts of width 1536; 235B total, 22B active per token
Mixture of experts in Qwen
Each Qwen MoE layer replaces the dense SwiGLU with small SwiGLU experts and a linear router (see Mixture of experts (MoE) and Sparse MoE routing). The router picks the top- experts per token and mixes their outputs with softmax weights. Qwen1.5-MoE and Qwen2-MoE also have an always-on shared expert whose output passes through a sigmoid gate. Its job is to hold common knowledge so that the routed experts can specialise. Qwen3 drops the shared expert and balances the experts with a load-balancing loss computed over the whole global batch rather than per micro-batch, which allows experts to specialise by domain.
Worked example: one layer of Qwen3-30B-A3B
The config has , 48 layers, 32 query and 4 KV heads with , 128 experts of width 768 with 8 active, a 151,936-row vocabulary, and an untied head. Note that : Qwen3 sets the head size independently, so maps 2048 to 4096.
| part | formula | stored | used per token |
|---|---|---|---|
| W_q, W_o | 2 · 2048 · 4096 | 16,777,216 | 16,777,216 |
| W_k, W_v | 2 · 2048 · 512 | 4,194,304 | 4,194,304 |
| QK-Norm gains | 2 · 128 | 256 | 256 |
| router | 2048 · 128 | 262,144 | 262,144 |
| experts | 128 (or 8) · 3 · 2048 · 768 | 603,979,776 | 37,748,736 |
| 2 RMSNorm gains | 2 · 2048 | 4,096 | 4,096 |
| one layer | 623,120,640 | 56,889,600 | |
| × 48 + embeddings 622,329,856 + final norm | 30,532,122,624 | 3,353,032,704 |
From parameters to serving cost
- 1Experts are 96.9% of a layer's stored weights but only 8/128 of them run, so each layer stores 623.1M parameters and uses 56.9M. Over the model, 11.0% of the weights do the work for any one token.
- 2Compute per generated token: about the active matmul weights, GFLOP, which is close to what a dense 3B model costs.
- 3Memory: all 30.5B parameters must be resident, bytes GiB in bf16, close to what a dense 30B model needs.
- 4KV cache: bytes = 96 KiB per token, so a 32K context costs 3.00 GiB and a 128K context (YaRN ) 12.00 GiB.
- 5YaRN at with base , , : by the paper's ramp (, ), 24 of the 64 frequency pairs make more than 32 turns in 32K and are kept, 24 make less than one and are divided by 4, and 16 are blended. The logits are multiplied by . Implementations round the ramp boundaries, so the split can differ by a pair.
Qwen3: thinking and non-thinking in one model
Qwen3's post-trained models can do both. In thinking mode they write a reasoning trace inside <think> … </think> before the answer. In non-thinking mode they answer directly, and the chat template inserts an empty think block. The report describes four post-training stages for the flagship models: a long chain-of-thought cold start, reasoning RL, thinking-mode fusion (SFT on a mix of thinking and non-thinking data, so that one model learns both formats), and general RL. The smaller models are trained by strong-to-weak distillation from the large ones instead of running the full pipeline. A user switches modes with enable_thinking in the chat template, or per turn with /think and /no_think.
Because the two modes share weights, the report also describes a thinking budget. When the trace reaches a user-set number of tokens, generation is interrupted, a short instruction to stop thinking is inserted, and the model writes its answer from the partial trace. Accuracy rises smoothly with the budget, so a deployment can trade latency for quality per request. In July 2025 the Qwen team released updated "2507" checkpoints that split the instruct and thinking variants into separate models again.
1from transformers import AutoModelForCausalLM, AutoTokenizer23name = "Qwen/Qwen3-8B"4tok = AutoTokenizer.from_pretrained(name)5model = AutoModelForCausalLM.from_pretrained(name, torch_dtype="auto", device_map="auto")67def chat(prompt, thinking=True, budget=4096):8 text = tok.apply_chat_template(9 [{"role": "user", "content": prompt}],10 tokenize=False, add_generation_prompt=True,11 enable_thinking=thinking, # False: the template adds an empty <think></think>12 )13 ids = tok(text, return_tensors="pt").to(model.device)14 # model card: thinking T=0.6, top-p 0.95, top-k 20; non-thinking T=0.7, top-p 0.8, top-k 2015 out = model.generate(**ids, max_new_tokens=budget, do_sample=True,16 temperature=0.6 if thinking else 0.7,17 top_p=0.95 if thinking else 0.8, top_k=20)18 new = out[0][ids.input_ids.shape[1]:].tolist()19 try: # split at the last </think> (token id 151668)20 cut = len(new) - new[::-1].index(151668)21 except ValueError:22 cut = 023 return tok.decode(new[:cut], skip_special_tokens=True).strip(), \24 tok.decode(new[cut:], skip_special_tokens=True).strip()2526trace, answer = chat("How many positive divisors does 360 have?")27_, quick = chat("Translate 'good morning' into French.", thinking=False)28_, also_quick = chat("Is 2^31 - 1 prime? /no_think") # per-turn soft switch2930# Beyond the native 32K: static YaRN x4 (the scaling applies to every input,31# so the model card advises enabling it only when long inputs are expected)32long_model = AutoModelForCausalLM.from_pretrained(33 name, torch_dtype="auto", device_map="auto",34 rope_scaling={"rope_type": "yarn", "factor": 4.0, "original_max_position_embeddings": 32768},35 max_position_embeddings=131072,36)3738# Parameter count from the config (8.19B for Qwen3-8B, 6.95B without embeddings)39print(sum(p.numel() for p in model.parameters()) / 1e9)Use when
- +~152K byte-level BPE vocabulary: efficient on Chinese and many other languages, no unknown tokens
- +Full size ladder from 0.5B/0.6B to 72B dense and 235B MoE, many under Apache 2.0
- +GQA and a RoPE base of 10⁶ plus YaRN give 128K context at a modest KV cache (56 KiB/token for Qwen2.5-7B)
- +MoE models run 3B or 22B active parameters at the quality of much larger dense models
- +Qwen3: one set of weights for both fast answers and long reasoning, with a controllable thinking budget
Avoid when
- −Large vocabulary dominates small models (27.6% of Qwen2.5-0.5B even when tied)
- −QKV bias (Qwen–Qwen2.5) and QK-Norm (Qwen3) mean a plain Llama implementation is not weight-compatible
- −MoE saves compute, not memory: Qwen3-235B-A22B needs about 438 GiB of bf16 weights
- −Static YaRN in common runtimes degrades short-context quality unless switched on only when needed
- −Thinking mode multiplies output length, latency and KV-cache use
Check yourself
Qwen3-30B-A3B has 30.5B parameters but '3B active'. What does serving it cost, compared with a dense model?
Check yourself
What did Qwen3 change in the attention projections relative to Qwen2.5?
Remember
- ✓Qwen is a Llama-style decoder (pre-RMSNorm, SwiGLU, RoPE, untied head) with one ~152K byte-level BPE vocabulary (151,643 regular tokens) across all generations.
- ✓Attention: QKV bias in Qwen–Qwen2.5, replaced by per-head QK-Norm in Qwen3; MHA in Qwen, GQA in Qwen1.5's largest models and at every size from Qwen2.
- ✓Long context: RoPE base during 32K pre-training, then YaRN factor 4 (logits ) and Dual Chunk Attention to reach about 128K.
- ✓Data grew from 2–3T (Qwen) to 7T (Qwen2), 18T (Qwen2.5) and 36T tokens in 119 languages (Qwen3); MoE variants range from Qwen1.5-MoE-A2.7B to Qwen3-235B-A22B (128 experts, 8 active).
- ✓Qwen3 fuses thinking and non-thinking modes in one model, switched by enable_thinking or /think and /no_think, with a thinking budget.