# On-device generative AI

> On-device generation runs a 1–8B model on the user's phone or laptop: 4-bit weights plus KV cache must fit the RAM budget, and decode speed is capped by memory bandwidth ÷ bytes read per token.

Source: https://www.mlfoundry.in/learn/on-device-generative-ai  
Section: Generative AI › Serving Generative Models  
Level: advanced · 8 min read · updated 2026-10-21

**On-device generative AI:** means running a generative model's inference on the end user's hardware, such as a phone, laptop, car or browser, instead of on a server. Shipped examples include Google's Gemini Nano on the Pixel 8 Pro (December 2023) and Apple's roughly 3B-parameter on-device foundation model (June 2024). Open runtimes such as llama.cpp, with its GGUF file format, run open-weight models locally.

The reasons to move generation onto the device are concrete. Data never leaves it, which matters for messages, health and enterprise documents (see **Privacy and data leakage**). It works offline. There is no network round trip and no queue. The marginal cost per request is zero for the developer, since the user's hardware pays. The price is a much smaller model, so on-device models handle narrow tasks: smart replies, summarising a notification, rewriting a sentence, transcription, autocomplete, small image edits.

Three resources bind. **Memory**: the weights and the **KV cache** must fit in the RAM the operating system will give one app, alongside everything else running. **Memory bandwidth**: every decode step reads all the weights, so tokens per second are bounded by bandwidth divided by model bytes. **Energy and heat**: sustained generation drains the battery, and phones throttle when hot. Download size matters too: a multi-gigabyte model is a separate download that not every user will wait for.

## Will it fit, and how fast will it run?

$$
M=\underbrace{\frac{P\,b_\text{eff}}{8}}_{\text{weights}}+\underbrace{\frac{2\,L\,n_\text{kv}\,d_h\,T\,b_\text{kv}}{8}}_{\text{KV cache}}+M_\text{rt}\;\le\;M_\text{budget},\qquad b_\text{eff}=b+\frac{16}{g}
$$

- $P,\;b$: parameter count and bits per weight
- $g$: quantization group size; each group stores one fp16 scale, adding $16/g$ bits per weight
- $L,\;n_\text{kv},\;d_h$: layers, key/value heads and head dimension; the 2 counts keys and values
- $T,\;b_\text{kv}$: context length in tokens and bits per cached value
- $M_\text{rt}$: runtime buffers and activations (0.25 GB in the examples)

$$
\text{tokens/s}\;\le\;\frac{\text{memory bandwidth}}{P\,b_\text{eff}/8+\text{KV bytes}},\qquad t_\text{prefill}\approx\frac{2\,P\,N_\text{prompt}}{\text{sustained FLOP/s}}
$$

- $\text{decode}$: one token per step, with all weights read each step; memory-bound on every phone and laptop
- $\text{prefill}$: the whole prompt in parallel, about $2P$ FLOPs per token; compute-bound, a job for the GPU or NPU

Quantization therefore does two things at once. It shrinks the weights enough to load them, and it speeds up decoding in proportion to the bytes saved. Prefill is the other bottleneck for long prompts: a 3.21B model needs about $2\times3.21\times10^9\times1{,}000=6.42$ TFLOP for a 1,000-token prompt, roughly 1.3 s at an illustrative sustained 5 TFLOP/s. That is why on-device features keep prompts short.

> Figure: A 3B model on an 8 GB phone that gives the app about 4 GB. The weights at 4 bits with one fp16 scale per 32 weights (4.5 bits/weight) take 1.81 GB. A 4,096-token fp16 KV cache takes 0.47 GB (2 × 28 layers × 8 KV heads × 128 dims × 2 bytes = 112 KiB per token), and the runtime 0.25 GB, leaving 1.47 GB of headroom. At about 60 GB/s, decoding is bounded at 60 ÷ 2.28 ≈ 26 tokens/s. A router keeps hard requests on a server model.

## How models are made small enough

The standard compression toolbox, each covered in depth in the Deep Learning branch

| Technique | What it changes | Typical on-device setting |
| --- | --- | --- |
| **Quantization** | bits per weight (and per activation or KV entry) | 4-bit weights in groups of 32–128; int8 or int4 kernels on NPUs |
| **Knowledge distillation** | trains a small student on a large teacher's outputs | Gemini Nano was distilled from larger Gemini models |
| Small-model architecture | depth, width, attention sharing | grouped-query attention, shared input/output embeddings, deep-and-thin layouts (MobileLLM) |
| **Pruning** | removes weights or whole structures | structured pruning, so dense kernels still apply |
| **LoRA** adapters | one base model, many small task deltas | Apple ships task adapters over a single on-device base model |
| **Speculative decoding** | a tiny draft model proposes tokens, the main model verifies | cuts bandwidth-bound steps when the draft is accurate |

**Reported specifics (dated; on-device products change quickly)**

- **Gemini Nano (Dec 2023):** Nano-1 1.8B and Nano-2 3.25B parameters, distilled from larger Gemini models and quantized to 4 bits (Gemini technical report)
- **Phi-3-mini (Apr 2024):** 3.8B parameters; quantized to 4 bits it takes about 1.8 GB and ran natively on an iPhone 14 (A16) at more than 12 tokens/s, per the Phi-3 report
- **Apple (Jun 2024):** a ~3B on-device model with low-bit quantization and task-specific LoRA adapters; larger requests go to Private Cloud Compute servers
- **GGUF Q4_0 / Q8_0:** blocks of 32 weights with one fp16 scale: 4.5 and 8.5 bits per weight
- **LLM in a flash:** Alizadeh et al. (Apple, 2023) stream weights from flash on demand to run models up to twice the size of available DRAM

## Fit, speed and quantization error

> Figure: Defaults (3B, 4-bit, groups of 32, 4,096-token fp16 KV cache): 4.50 bits/weight and 2.53 GB in total. That fits the phone's 4 GB budget with decode bounded at 26.4 tok/s (laptop 52.7, workstation 175.8). On the synthetic layer the output error is 14.43% at 4 bits, 28.63% at 3 bits and 1.02% at 8 bits. With one group per 512-weight row, 4-bit error jumps to 44.11%, because outliers set the scale. Raise the context to 32,768 and the KV cache grows to 3.76 GB, so the 3B model (5.81 GB) no longer fits the phone.

The synthetic layer has 1% outlier weights, eight times larger than the rest, as real LLM layers do. With absmax scaling one outlier stretches the scale for its whole group, so the other weights are squeezed into a few levels. Small groups contain the damage, at $16/g$ extra bits per weight. Real 4-bit schemes add calibration (GPTQ, AWQ) or mixed precision on top; see **Quantization** and **QLoRA**.

## Worked example: a 3B model on a phone

3.21B parameters, Llama 3.2 3B shapes (28 layers, 8 KV heads × 128 dims), 4 GB app budget, 60 GB/s, runtime 0.25 GB

| Setting | Weights | KV cache | Total | Fits? | Decode bound |
| --- | --- | --- | --- | --- | --- |
| fp16, 4k context | 3.21 × 2 = 6.42 GB | 0.47 GB | 7.14 GB | no | 60 ÷ 6.89 = 8.7 tok/s |
| 8-bit (8.5 bpw), 4k | 3.41 GB | 0.47 GB | 4.13 GB | no, just over | 15.5 tok/s |
| 4-bit (4.5 bpw), 4k | 1.81 GB | 0.47 GB | 2.53 GB | yes | 60 ÷ 2.28 = 26.4 tok/s |
| 4-bit + 8-bit KV, 4k | 1.81 GB | 0.23 GB | 2.29 GB | yes | 29.4 tok/s |
| 4-bit, 32k context | 1.81 GB | 3.76 GB | 5.81 GB | no | 10.8 tok/s |

The bound is optimistic: real kernels reach only part of the peak bandwidth, and the phone throttles under sustained load. Still, the ordering is reliable. Bits per weight decide whether the model loads at all. At long contexts the KV cache, not the weights, becomes the binding term.

```python
def footprint_gb(params, bits, group=32, ctx=4096, layers=28, kv_heads=8, head_dim=128, kv_bits=16):
    bpw = 16 if bits >= 16 else bits + 16 / group           # + one fp16 scale per group
    weights = params * bpw / 8 / 1e9
    kv = ctx * 2 * layers * kv_heads * head_dim * kv_bits / 8 / 1e9
    return weights, kv

w, kv = footprint_gb(3.21e9, bits=4)                       # 3B model, Llama 3.2 shapes
print(round(w, 2), round(kv, 2), round(60 / (w + kv), 1))  # 1.81 0.47 26.4  (60 GB/s phone)

from llama_cpp import Llama                                 # pip install llama-cpp-python
llm = Llama(model_path="model-3b-instruct.Q4_0.gguf", n_ctx=4096, n_gpu_layers=-1)
out = llm.create_chat_completion(
    messages=[{"role": "user", "content": "Suggest a short reply to: Running 10 min late?"}],
    max_tokens=32, temperature=0.3)
print(out["choices"][0]["message"]["content"])
```

Memory and speed estimate (runs as is), then local inference with llama-cpp-python (not run here; the GGUF file name is a placeholder)

> **Benchmarks on a desktop, deployment on a phone:** Quality drops from quantization are usually measured as perplexity on a GPU. On the phone you also have a different kernel, a shorter context, a smaller sampling budget and thermal throttling. Measure task quality and tokens per second on the slowest device you support, after several minutes of sustained load, not on the first request.

> **Hybrid is the common design:** Production systems rarely run everything locally. A small local model handles the frequent, short, private requests. A router sends long-context or hard requests to a server model, with the user's consent where data leaves the device. The trade-offs mirror those in **Cost and latency optimization**: the local path costs nothing per call and has no network latency, but answers fewer questions well.

**Strengths**

- Private by construction: prompts and outputs can stay on the device.
- Works offline, with no network round trip or server queue.
- Zero marginal serving cost for the provider; scales with the user base for free.

**Limitations**

- Model size is capped at a few billion parameters, so reasoning and knowledge are limited.
- Hardware is fragmented: different NPUs, kernels and memory limits per device class.
- Battery, heat and download size constrain how often and how long it can run; updates ship as app or OS updates.

**Question:** An 8B model is quantized from 4.5 to 3.5 bits per weight. Roughly what happens to its decode speed bound on the same device at short context?

- A. Nothing; decode is compute-bound
- B. It rises by about 4.5 / 3.5 ≈ 1.29×
- C. It doubles
- D. It falls, because dequantization costs time

**Answer:** B. It rises by about 4.5 / 3.5 ≈ 1.29× — At short context the bytes read per step are dominated by the weights: $8.03\times10^9\times4.5/8=4.52$ GB versus $3.51$ GB. The bound is bandwidth divided by bytes, so it rises by $4.52/3.51\approx1.29$. Dequantization is cheap next to the memory traffic, though real kernels do not reach the full bound.

**Question:** A 3B model with a 4,096-token context fits on a phone. The product team asks for a 32,768-token context. What breaks first?

- A. The weights, which grow with context
- B. The KV cache, which grows linearly with context: 0.47 GB becomes 3.76 GB
- C. The tokenizer vocabulary
- D. Nothing, because attention is linear in context

**Answer:** B. The KV cache, which grows linearly with context: 0.47 GB becomes 3.76 GB — Weights are fixed, but the KV cache holds 112 KiB per token for this shape. Eight times the context means eight times the cache. That alone nearly fills the 4 GB budget, and the total becomes 5.81 GB. KV-cache quantization, sliding-window attention or a shorter context are the usual remedies.

**Key points**

- On-device generation trades model size for privacy, offline use, no network latency and zero serving cost.
- Memory: $P\,b_\text{eff}/8$ weights plus a KV cache of $2Ln_\text{kv}d_hT\,b_\text{kv}/8$ bytes must fit the per-app budget.
- Decode is memory-bandwidth-bound: tokens/s ≤ bandwidth ÷ bytes read per token, so fewer bits mean both fit and speed.
- Group-wise 4-bit quantization (about 4.5 bits/weight at g = 32) is the usual default; small groups contain outlier damage.
- At long contexts the KV cache, not the weights, becomes the limit.
- Real deployments are hybrid: a small local model with LoRA adapters, plus a server model for hard requests.

---

From MLFoundry (https://www.mlfoundry.in). The web page adds interactive figures and diagrams.
