Definition
On-device generative AI means running a generative model's inference on the end user's hardware, such as a phone, laptop, car or browser, instead of on a server. Shipped examples include Google's Gemini Nano on the Pixel 8 Pro (December 2023) and Apple's roughly 3B-parameter on-device foundation model (June 2024). Open runtimes such as llama.cpp, with its GGUF file format, run open-weight models locally.
The reasons to move generation onto the device are concrete. Data never leaves it, which matters for messages, health and enterprise documents (see Privacy and data leakage). It works offline. There is no network round trip and no queue. The marginal cost per request is zero for the developer, since the user's hardware pays. The price is a much smaller model, so on-device models handle narrow tasks: smart replies, summarising a notification, rewriting a sentence, transcription, autocomplete, small image edits.
Three resources bind. Memory: the weights and the KV cache must fit in the RAM the operating system will give one app, alongside everything else running. Memory bandwidth: every decode step reads all the weights, so tokens per second are bounded by bandwidth divided by model bytes. Energy and heat: sustained generation drains the battery, and phones throttle when hot. Download size matters too: a multi-gigabyte model is a separate download that not every user will wait for.
Will it fit, and how fast will it run?
- parameter count and bits per weight
- quantization group size; each group stores one fp16 scale, adding bits per weight
- layers, key/value heads and head dimension; the 2 counts keys and values
- context length in tokens and bits per cached value
- runtime buffers and activations (0.25 GB in the examples)
- one token per step, with all weights read each step; memory-bound on every phone and laptop
- the whole prompt in parallel, about FLOPs per token; compute-bound, a job for the GPU or NPU
Quantization therefore does two things at once. It shrinks the weights enough to load them, and it speeds up decoding in proportion to the bytes saved. Prefill is the other bottleneck for long prompts: a 3.21B model needs about TFLOP for a 1,000-token prompt, roughly 1.3 s at an illustrative sustained 5 TFLOP/s. That is why on-device features keep prompts short.
How models are made small enough
| Technique | What it changes | Typical on-device setting |
|---|---|---|
| Quantization | bits per weight (and per activation or KV entry) | 4-bit weights in groups of 32–128; int8 or int4 kernels on NPUs |
| Knowledge distillation | trains a small student on a large teacher's outputs | Gemini Nano was distilled from larger Gemini models |
| Small-model architecture | depth, width, attention sharing | grouped-query attention, shared input/output embeddings, deep-and-thin layouts (MobileLLM) |
| Pruning | removes weights or whole structures | structured pruning, so dense kernels still apply |
| LoRA adapters | one base model, many small task deltas | Apple ships task adapters over a single on-device base model |
| Speculative decoding | a tiny draft model proposes tokens, the main model verifies | cuts bandwidth-bound steps when the draft is accurate |
Reported specifics (dated; on-device products change quickly)
- Gemini Nano (Dec 2023)
- Nano-1 1.8B and Nano-2 3.25B parameters, distilled from larger Gemini models and quantized to 4 bits (Gemini technical report)
- Phi-3-mini (Apr 2024)
- 3.8B parameters; quantized to 4 bits it takes about 1.8 GB and ran natively on an iPhone 14 (A16) at more than 12 tokens/s, per the Phi-3 report
- Apple (Jun 2024)
- a ~3B on-device model with low-bit quantization and task-specific LoRA adapters; larger requests go to Private Cloud Compute servers
- GGUF Q4_0 / Q8_0
- blocks of 32 weights with one fp16 scale: 4.5 and 8.5 bits per weight
- LLM in a flash
- Alizadeh et al. (Apple, 2023) stream weights from flash on demand to run models up to twice the size of available DRAM
Fit, speed and quantization error
The synthetic layer has 1% outlier weights, eight times larger than the rest, as real LLM layers do. With absmax scaling one outlier stretches the scale for its whole group, so the other weights are squeezed into a few levels. Small groups contain the damage, at extra bits per weight. Real 4-bit schemes add calibration (GPTQ, AWQ) or mixed precision on top; see Quantization and QLoRA.
Worked example: a 3B model on a phone
| Setting | Weights | KV cache | Total | Fits? | Decode bound |
|---|---|---|---|---|---|
| fp16, 4k context | 3.21 × 2 = 6.42 GB | 0.47 GB | 7.14 GB | no | 60 ÷ 6.89 = 8.7 tok/s |
| 8-bit (8.5 bpw), 4k | 3.41 GB | 0.47 GB | 4.13 GB | no, just over | 15.5 tok/s |
| 4-bit (4.5 bpw), 4k | 1.81 GB | 0.47 GB | 2.53 GB | yes | 60 ÷ 2.28 = 26.4 tok/s |
| 4-bit + 8-bit KV, 4k | 1.81 GB | 0.23 GB | 2.29 GB | yes | 29.4 tok/s |
| 4-bit, 32k context | 1.81 GB | 3.76 GB | 5.81 GB | no | 10.8 tok/s |
The bound is optimistic: real kernels reach only part of the peak bandwidth, and the phone throttles under sustained load. Still, the ordering is reliable. Bits per weight decide whether the model loads at all. At long contexts the KV cache, not the weights, becomes the binding term.
1def footprint_gb(params, bits, group=32, ctx=4096, layers=28, kv_heads=8, head_dim=128, kv_bits=16):2 bpw = 16 if bits >= 16 else bits + 16 / group # + one fp16 scale per group3 weights = params * bpw / 8 / 1e94 kv = ctx * 2 * layers * kv_heads * head_dim * kv_bits / 8 / 1e95 return weights, kv67w, kv = footprint_gb(3.21e9, bits=4) # 3B model, Llama 3.2 shapes8print(round(w, 2), round(kv, 2), round(60 / (w + kv), 1)) # 1.81 0.47 26.4 (60 GB/s phone)910from llama_cpp import Llama # pip install llama-cpp-python11llm = Llama(model_path="model-3b-instruct.Q4_0.gguf", n_ctx=4096, n_gpu_layers=-1)12out = llm.create_chat_completion(13 messages=[{"role": "user", "content": "Suggest a short reply to: Running 10 min late?"}],14 max_tokens=32, temperature=0.3)15print(out["choices"][0]["message"]["content"])Use when
- +Private by construction: prompts and outputs can stay on the device.
- +Works offline, with no network round trip or server queue.
- +Zero marginal serving cost for the provider; scales with the user base for free.
Avoid when
- −Model size is capped at a few billion parameters, so reasoning and knowledge are limited.
- −Hardware is fragmented: different NPUs, kernels and memory limits per device class.
- −Battery, heat and download size constrain how often and how long it can run; updates ship as app or OS updates.
Check yourself
An 8B model is quantized from 4.5 to 3.5 bits per weight. Roughly what happens to its decode speed bound on the same device at short context?
Check yourself
A 3B model with a 4,096-token context fits on a phone. The product team asks for a 32,768-token context. What breaks first?
Remember
- ✓On-device generation trades model size for privacy, offline use, no network latency and zero serving cost.
- ✓Memory: weights plus a KV cache of bytes must fit the per-app budget.
- ✓Decode is memory-bandwidth-bound: tokens/s ≤ bandwidth ÷ bytes read per token, so fewer bits mean both fit and speed.
- ✓Group-wise 4-bit quantization (about 4.5 bits/weight at g = 32) is the usual default; small groups contain outlier damage.
- ✓At long contexts the KV cache, not the weights, becomes the limit.
- ✓Real deployments are hybrid: a small local model with LoRA adapters, plus a server model for hard requests.