On-device generative AI

On-device generation runs a 1–8B model on the user's phone or laptop: 4-bit weights plus KV cache must fit the RAM budget, and decode speed is capped by memory bandwidth ÷ bytes read per token.

8 min read611 wordsadvanced0 viewsupdated 2026-10-21

Definition

On-device generative AI means running a generative model's inference on the end user's hardware, such as a phone, laptop, car or browser, instead of on a server. Shipped examples include Google's Gemini Nano on the Pixel 8 Pro (December 2023) and Apple's roughly 3B-parameter on-device foundation model (June 2024). Open runtimes such as llama.cpp, with its GGUF file format, run open-weight models locally.

The reasons to move generation onto the device are concrete. Data never leaves it, which matters for messages, health and enterprise documents (see Privacy and data leakage). It works offline. There is no network round trip and no queue. The marginal cost per request is zero for the developer, since the user's hardware pays. The price is a much smaller model, so on-device models handle narrow tasks: smart replies, summarising a notification, rewriting a sentence, transcription, autocomplete, small image edits.

Three resources bind. Memory: the weights and the KV cache must fit in the RAM the operating system will give one app, alongside everything else running. Memory bandwidth: every decode step reads all the weights, so tokens per second are bounded by bandwidth divided by model bytes. Energy and heat: sustained generation drains the battery, and phones throttle when hot. Download size matters too: a multi-gigabyte model is a separate download that not every user will wait for.

Will it fit, and how fast will it run?

M=P beff8⏟weights+2 L nkv dh T bkv8⏟KV cache+Mrt  ≤  Mbudget,beff=b+16gM=\underbrace{\frac{P\,b_\text{eff}}{8}}_{\text{weights}}+\underbrace{\frac{2\,L\,n_\text{kv}\,d_h\,T\,b_\text{kv}}{8}}_{\text{KV cache}}+M_\text{rt}\;\le\;M_\text{budget},\qquad b_\text{eff}=b+\frac{16}{g}
P,  bP,\;b
parameter count and bits per weight
gg
quantization group size; each group stores one fp16 scale, adding 16/g16/g bits per weight
L,  nkv,  dhL,\;n_\text{kv},\;d_h
layers, key/value heads and head dimension; the 2 counts keys and values
T,  bkvT,\;b_\text{kv}
context length in tokens and bits per cached value
MrtM_\text{rt}
runtime buffers and activations (0.25 GB in the examples)
tokens/s  ≤  memory bandwidthP beff/8+KV bytes,tprefill≈2 P Npromptsustained FLOP/s\text{tokens/s}\;\le\;\frac{\text{memory bandwidth}}{P\,b_\text{eff}/8+\text{KV bytes}},\qquad t_\text{prefill}\approx\frac{2\,P\,N_\text{prompt}}{\text{sustained FLOP/s}}
decode\text{decode}
one token per step, with all weights read each step; memory-bound on every phone and laptop
prefill\text{prefill}
the whole prompt in parallel, about 2P2P FLOPs per token; compute-bound, a job for the GPU or NPU

Quantization therefore does two things at once. It shrinks the weights enough to load them, and it speeds up decoding in proportion to the bytes saved. Prefill is the other bottleneck for long prompts: a 3.21B model needs about 2×3.21×109×1,000=6.422\times3.21\times10^9\times1{,}000=6.42 TFLOP for a 1,000-token prompt, roughly 1.3 s at an illustrative sustained 5 TFLOP/s. That is why on-device features keep prompts short.

a 3B model on an 8 GB phoneapp: smart replyin: "Running 10 min late?"out: "No worries, see you soon"local runtimellama.cpp · Core ML · ExecuTorchONNX Runtime · MediaPipe / LiteRTmodel file (shipped or downloaded)3.21B weights, 4-bit, groups of 32+ fp16 scale = 4.5 bits → 1.81 GB+ LoRA adapter per task (MBs)CPUdecodeGPUprefillNPUint8/int4shared LPDDR memory, ~60 GB/s8 GB RAM, budget 4 GBOS + other apps 4.00 GBfree headroom 1.47 GBruntime 0.25 GBKV cache 0.47 GBweights 1.81 GB4 GB08KV: 2 × 28 layers × 8 heads × 128 × 2 B= 112 KiB/token × 4,096 = 0.47 GBdecode bound: 60 GB/s ÷ (1.81 + 0.47) GB read per token ≈ 26 tok/srequestshort, private,offline-capable?yes → on-device modelno → server modelno network, data stays locallarger model, costs per call
fig.A 3B model on an 8 GB phone that gives the app about 4 GB. The weights at 4 bits with one fp16 scale per 32 weights (4.5 bits/weight) take 1.81 GB. A 4,096-token fp16 KV cache takes 0.47 GB (2 × 28 layers × 8 KV heads × 128 dims × 2 bytes = 112 KiB per token), and the runtime 0.25 GB, leaving 1.47 GB of headroom. At about 60 GB/s, decoding is bounded at 60 ÷ 2.28 ≈ 26 tokens/s. A router keeps hard requests on a server model.

How models are made small enough

TechniqueWhat it changesTypical on-device setting
Quantizationbits per weight (and per activation or KV entry)4-bit weights in groups of 32–128; int8 or int4 kernels on NPUs
Knowledge distillationtrains a small student on a large teacher's outputsGemini Nano was distilled from larger Gemini models
Small-model architecturedepth, width, attention sharinggrouped-query attention, shared input/output embeddings, deep-and-thin layouts (MobileLLM)
Pruningremoves weights or whole structuresstructured pruning, so dense kernels still apply
LoRA adaptersone base model, many small task deltasApple ships task adapters over a single on-device base model
Speculative decodinga tiny draft model proposes tokens, the main model verifiescuts bandwidth-bound steps when the draft is accurate
The standard compression toolbox, each covered in depth in the Deep Learning branch

Reported specifics (dated; on-device products change quickly)

Gemini Nano (Dec 2023)
Nano-1 1.8B and Nano-2 3.25B parameters, distilled from larger Gemini models and quantized to 4 bits (Gemini technical report)
Phi-3-mini (Apr 2024)
3.8B parameters; quantized to 4 bits it takes about 1.8 GB and ran natively on an iPhone 14 (A16) at more than 12 tokens/s, per the Phi-3 report
Apple (Jun 2024)
a ~3B on-device model with low-bit quantization and task-specific LoRA adapters; larger requests go to Private Cloud Compute servers
GGUF Q4_0 / Q8_0
blocks of 32 weights with one fp16 scale: 4.5 and 8.5 bits per weight
LLM in a flash
Alizadeh et al. (Apple, 2023) stream weights from flash on demand to run models up to twice the size of available DRAM

Fit, speed and quantization error

3B model (3.21B params) · 4-bit, group 32 = 4.50 bits/weight · ctx 4096 · KV 16-bitweights 1.81 GB + KV cache 0.47 GB (112 KiB/token × 4096) + runtime 0.25 GB = 2.53 GBphone8 GB, 60 GB/s2.53 / 4 GB budget · fits≤ 26.4 tok/sdecode boundlaptop16 GB, 120 GB/s2.53 / 10 GB budget · fits≤ 52.7 tok/sdecode boundworkstation64 GB, 400 GB/s2.53 / 48 GB budget · fits≤ 175.8 tok/sdecode boundweightsKV cacheruntime| budgetmeasured: ‖Wx − Ŵx‖ / ‖Wx‖ for a 64×512 synthetic layer (1% outlier weights), absmax groups of 3247.03%2-bit2.50 bpw28.63%3-bit3.50 bpw14.43%4-bit4.50 bpw1.02%8-bit8.50 bpwcurrent: 14.43% errorsmaller groups: lesserror, more scale bitsdecode bound = bandwidth ÷ (weights + KV bytes) · devices illustrative · Llama 3.x shapes
modelKV cache
weights
group size
fig.Defaults (3B, 4-bit, groups of 32, 4,096-token fp16 KV cache): 4.50 bits/weight and 2.53 GB in total. That fits the phone's 4 GB budget with decode bounded at 26.4 tok/s (laptop 52.7, workstation 175.8). On the synthetic layer the output error is 14.43% at 4 bits, 28.63% at 3 bits and 1.02% at 8 bits. With one group per 512-weight row, 4-bit error jumps to 44.11%, because outliers set the scale. Raise the context to 32,768 and the KV cache grows to 3.76 GB, so the 3B model (5.81 GB) no longer fits the phone.

The synthetic layer has 1% outlier weights, eight times larger than the rest, as real LLM layers do. With absmax scaling one outlier stretches the scale for its whole group, so the other weights are squeezed into a few levels. Small groups contain the damage, at 16/g16/g extra bits per weight. Real 4-bit schemes add calibration (GPTQ, AWQ) or mixed precision on top; see Quantization and QLoRA.

Worked example: a 3B model on a phone

SettingWeightsKV cacheTotalFits?Decode bound
fp16, 4k context3.21 × 2 = 6.42 GB0.47 GB7.14 GBno60 ÷ 6.89 = 8.7 tok/s
8-bit (8.5 bpw), 4k3.41 GB0.47 GB4.13 GBno, just over15.5 tok/s
4-bit (4.5 bpw), 4k1.81 GB0.47 GB2.53 GByes60 ÷ 2.28 = 26.4 tok/s
4-bit + 8-bit KV, 4k1.81 GB0.23 GB2.29 GByes29.4 tok/s
4-bit, 32k context1.81 GB3.76 GB5.81 GBno10.8 tok/s
3.21B parameters, Llama 3.2 3B shapes (28 layers, 8 KV heads × 128 dims), 4 GB app budget, 60 GB/s, runtime 0.25 GB

The bound is optimistic: real kernels reach only part of the peak bandwidth, and the phone throttles under sustained load. Still, the ordering is reliable. Bits per weight decide whether the model loads at all. At long contexts the KV cache, not the weights, becomes the binding term.

python
1def footprint_gb(params, bits, group=32, ctx=4096, layers=28, kv_heads=8, head_dim=128, kv_bits=16):
2 bpw = 16 if bits >= 16 else bits + 16 / group # + one fp16 scale per group
3 weights = params * bpw / 8 / 1e9
4 kv = ctx * 2 * layers * kv_heads * head_dim * kv_bits / 8 / 1e9
5 return weights, kv
6
7w, kv = footprint_gb(3.21e9, bits=4) # 3B model, Llama 3.2 shapes
8print(round(w, 2), round(kv, 2), round(60 / (w + kv), 1)) # 1.81 0.47 26.4 (60 GB/s phone)
9
10from llama_cpp import Llama # pip install llama-cpp-python
11llm = Llama(model_path="model-3b-instruct.Q4_0.gguf", n_ctx=4096, n_gpu_layers=-1)
12out = llm.create_chat_completion(
13 messages=[{"role": "user", "content": "Suggest a short reply to: Running 10 min late?"}],
14 max_tokens=32, temperature=0.3)
15print(out["choices"][0]["message"]["content"])
Memory and speed estimate (runs as is), then local inference with llama-cpp-python (not run here; the GGUF file name is a placeholder)

Use when

  • +Private by construction: prompts and outputs can stay on the device.
  • +Works offline, with no network round trip or server queue.
  • +Zero marginal serving cost for the provider; scales with the user base for free.

Avoid when

  • −Model size is capped at a few billion parameters, so reasoning and knowledge are limited.
  • −Hardware is fragmented: different NPUs, kernels and memory limits per device class.
  • −Battery, heat and download size constrain how often and how long it can run; updates ship as app or OS updates.

Check yourself

An 8B model is quantized from 4.5 to 3.5 bits per weight. Roughly what happens to its decode speed bound on the same device at short context?

Check yourself

A 3B model with a 4,096-token context fits on a phone. The product team asks for a 32,768-token context. What breaks first?

Remember

  • ✓On-device generation trades model size for privacy, offline use, no network latency and zero serving cost.
  • ✓Memory: P beff/8P\,b_\text{eff}/8 weights plus a KV cache of 2LnkvdhT bkv/82Ln_\text{kv}d_hT\,b_\text{kv}/8 bytes must fit the per-app budget.
  • ✓Decode is memory-bandwidth-bound: tokens/s ≤ bandwidth ÷ bytes read per token, so fewer bits mean both fit and speed.
  • ✓Group-wise 4-bit quantization (about 4.5 bits/weight at g = 32) is the usual default; small groups contain outlier damage.
  • ✓At long contexts the KV cache, not the weights, becomes the limit.
  • ✓Real deployments are hybrid: a small local model with LoRA adapters, plus a server model for hard requests.
← back to Generative AI roadmap