Privacy as architecture: the on-device LLM turn

Most privacy claims are promises. We won’t look, we won’t sell, we’ll delete it later. They rely on trust because the data still leaves your device and lands on someone else’s server. On-device inference changes the category of the claim entirely: if the model runs locally, the sensitive data physically never goes anywhere, so there is nothing left to promise about. Privacy becomes a property of the architecture, not a line in a policy.

The reason this is suddenly practical is that small models got good. A 2–4B model running through MediaPipe or Ollama on a phone or laptop is now capable enough to be useful, and work like NVIDIA’s Nemotron Nano 2 — a 9B hybrid Mamba-Transformer that matches larger models at up to 6× the throughput on a 128K context — shows the efficiency curve is still bending in the right direction. You no longer have to choose between “runs locally” and “actually works.”

What gets easier

Running the model on the device isn’t only a compliance win. Several hard problems quietly disappear:

• No round-trip latency. The model responds at the speed of local compute, not the network.
• Works offline. Care, safety, and accessibility tools can’t depend on a signal that might not be there.
• No per-token cost. Inference is effectively free after the hardware is bought, which changes what you can afford to run continuously rather than on-demand.

What gets harder

It isn’t a free lunch. Small models are genuinely smaller — you design around their limits instead of assuming a frontier model will cover for you. Quantization, memory pressure, and thermals become product constraints rather than backend details. And you’re shipping updates to hardware you don’t control.

Where the constraint is the point

The projects I keep coming back to are the ones where “on-device” isn’t a limitation to apologize for — it’s the entire pitch.

• Weave makes personalized product-placement decisions with Gemma running locally through Ollama, behind a “privacy firewall” that meters every outbound byte and proves the count is zero. Per-viewer targeting normally requires shipping your profile to an ad network; here it can’t, by construction.
• Saathi is an offline companion for people living with Alzheimer’s, running Gemma on an Android phone. The whole point is that a family’s home life never becomes telemetry.
• AWARE interprets Wi-Fi sensing data at the edge and only lets events leave the home — never the raw signal.

In each of these, moving the model to the cloud wouldn’t just be worse for privacy — it would break the product. That’s the tell that on-device is the right call: not when it’s a nice-to-have, but when the guarantee is the feature.