Small models on the device: where they are enough, where the cloud still wins

What small models do well, and where they fall short

By small language model (SLM) we mean up to about 10 billion parameters; on a phone, 1 to 4 billion (Apple's on-device model has 3 billion). Such a model fits in a few gigabytes, needs no network and does well at:

• Classification and routing: intent, sentiment, language, which model takes a request.
• Extraction into a fixed format: fields from a receipt, an email or a form.
• Rewriting, short summaries and autocomplete: a notification, a review, a suggested reply.
• Small offline assistants: Microsoft's 330-million-parameter Mu model turns plain language into Windows Settings actions.

The limits show up fast. On the Artificial Analysis Intelligence Index (September 26, 2026), the best open models up to 4B score 6 to 16 points, against about 58 for the leading frontier model. Small models remember little, invent answers to open questions and plan poorly, so multi-step agents break.

What is available in September 2026

Built into the platform

No model to ship, no per-token bill; but you choose neither the model, nor its version, nor the devices that get it.

Open models you bundle

You pick the version and can fine-tune it; you pay in downloads, memory, updates and licenses.

All eight cover Spanish and French, at least on paper. Watch the licenses: Tiny Aya is non-commercial, and Liquid's LFM2 license allows free commercial use only below USD 10 million in revenue. Runtimes include Google's LiteRT-LM, Apple's Core AI and MLX, and Microsoft's Foundry Local (about 20 MB per app).

Hardware: NPUs, memory and the browser

TOPS sell laptops; memory and bandwidth decide what runs. Google puts Gemma 4 E2B at 2.9 GB in INT4 (11.4 GB in BF16), and Apple keeps its larger on-device model (AFM 3 Core Advanced, 20B sparse, 1 to 4B active) for its newest iPhones and for recent Macs and iPads with at least 12 GB. In Apple's MLX tests, generation ran 19 to 27% faster on M5 than on M4, in line with 28% more bandwidth. See our quantization guide.

In the browser, Chrome shares one Gemini Nano download across sites, on desktop only; Google's LiteRT.js runs your own model on CPU (WebAssembly), GPU (WebGPU) or, experimentally, NPU (WebNN), with a download per user. More in local AI hardware in 2026.

For: privacy and latency. Against: battery, memory, fragmentation

The strongest argument is legal. If data never leaves the device, there is no AI vendor to contract as a processor (GDPR Article 28) and no international transfer to justify (GDPR Chapter V, Article 26 of Colombia's Ley 1581). You still answer for what you process, but compliance gets simpler. On Android, Google says AICore has no direct internet access and keeps no record of inputs or outputs.

Latency and offline use follow: Microsoft reports over 100 tokens per second for Mu on Copilot+ NPUs; Liquid AI, 70 for LFM2.5-1.2B on a Galaxy S25 Ultra CPU, in 719 MB. Against that:

• Battery: Google measured 0.75% of a Pixel 9 Pro battery for 25 conversations with Gemma 3 270M in INT4; on a desktop GPU, a 7B model used up to 4.4 times more energy per token than a 1B one.
• Storage and context: Chrome needs 22 GB free for Gemini Nano and deletes it below 10 GB; Apple's model holds a few thousand tokens per session.
• Updates and coverage: Windows swaps Phi Silica for Aion Instruct in November 2026, the ML Kit Prompt API is a beta without an SLA and Android serves only the foreground app.

Specializing a small model: LoRA and distillation

Tuned for one task, a mediocre generalist can become good enough. LoRA trains small adapter matrices on frozen weights; distillation has a large model produce the examples a small one learns from. The tooling exists: mlx-lm supports LoRA on a Mac, and since 2025 Apple has offered a toolkit to train LoRA adapters for its on-device model. DeepSeek trained Qwen3-8B on R1-0528 reasoning traces and says it matches Qwen3-235B in thinking mode on AIME 2024.

1. Pick one narrow task with a pass/fail metric: right label, right field, valid JSON.
2. Build a test set of real, anonymized cases and measure the base model first; a good prompt may suffice.
3. Generate training data with a large model whose terms allow it, and have people review a sample.
4. Train on the exact base version you ship, quantize, and re-test on real devices: in one study, 4-bit weights kept 93.5% of reasoning accuracy at 1.5B, against 99.5% at 32B.
5. Ship behind a cloud fallback and retrain when the base model changes. If knowledge is what's missing, see fine-tuning or RAG.

Device, server or frontier API

Decide per feature, not per app.

The hybrid pattern: local first, cloud fallback

1. Check at run time that the local model is present and supports the user's language.
2. Route by task: narrow work stays local; open questions, long documents and agents go to the cloud.
3. Constrain the local output with structured generation (Apple's guided generation, JSON schemas in Chrome's Prompt API) and validate it.
4. Escalate when validation fails or the input exceeds the local context, if the user allows data to leave the device.

Apple now builds this in: since iOS 27, one session API reaches the on-device model, Private Cloud Compute (32,000 tokens; no API cost for developers under 2 million first-time downloads) or, through the new LanguageModel protocol, other models; Apple says Anthropic and Google will publish Swift packages for Claude and Gemini.

Rule of thumb: log the escalation rate, without content, by device and OS version. If it jumps after an OS update, the platform model has changed: re-run your evaluation.

What it costs: a method with explicit assumptions

Take a feature that classifies and summarizes short messages: 1 million tasks a month, 500 input and 100 output tokens each (illustrative assumptions, not measurements).

Against an open model served by API, the on-device saving is negligible; against a frontier model it is real but rarely decisive. The deciding costs are engineering and testing per platform, fallback traffic and downloads (2.9 GB per user for Gemma 4 E2B in INT4, at every update). The same logic sets break-even: at $18 a month, a $2,000 GPU would take over nine years to pay back. More in the real cost of AI.

What it means in Colombia, Latin America and Europe

Colombia and Latin America: design for the real device mix. Built-in models reach only recent phones (Pixel 9, Galaxy S25, iPhone 15 Pro and later); mid-range and older devices need another path. For mobile apps used without coverage in field sales, logistics or agriculture, a bundled 1B model (about 1 GB in 4-bit) can extract data offline. Avoid multi-gigabyte downloads over mobile data: Chrome itself waits for an unmetered connection.

Colombia: fewer data questions. The SIC applies Ley 1581 to AI (Circular Externa 002 of 2024). The United States is on its adequacy list, so a US API is not prohibited, but you must demonstrate accountability; on the device, that question disappears.

Europe: minimization by design. Nothing reaches your servers unless you decide so, and a LoRA adapter does not make you a general-purpose AI model provider under the AI Act: the Commission's guidelines set that bar at over a third of the original training compute.