Choosing the right LLM for on-device inference

By Ryze AI Team

On-device inference changes every assumption you bring from cloud development. In the cloud you rent a GPU, pay per token, and reach for the largest model that fits your budget. On a phone, a browser tab, or an edge box, the constraints are memory, battery, and latency, and the smartest model is rarely the right one. A 22 million parameter model that answers reliably in 40 milliseconds will beat a 7 billion parameter model that needs four seconds and a gigabyte of RAM, every time, for the tasks where it is good enough. The work of choosing is mostly the work of admitting which tasks are small.

Start with the task, not the model

Before you look at a single leaderboard, write down the exact input and the exact output. Is the task classification, extraction, summarization, or open-ended generation? Each favors a different architecture. An encoder such as MiniLM is excellent at classification, semantic similarity, and embedding, and cannot generate a sentence. A small decoder such as a 135 million parameter Llama variant can generate prose but is weaker at long reasoning and prone to drift. Many teams waste weeks trying to make a decoder do classification well when an encoder would have solved it in an afternoon with higher accuracy and a tenth of the memory.

Also decide whether you need the model at all. A regex, a rules engine, or a small trained classifier often outperforms an LLM on narrow tasks and runs in microseconds. Reserve the LLM for problems where fuzzy understanding genuinely matters.

Parameter count and memory

Memory is the first hard wall. As a rule of thumb, a model needs roughly two bytes per parameter in FP16, one byte in INT8, and half a byte in INT4. A 500 million parameter model is about a gigabyte in FP16 and 250 megabytes in INT4. That sounds small until you add activations, the tokenizer, and the runtime overhead, which can double the footprint. Mobile and browser targets usually want under 200 megabytes total, which is why INT4 quantization is the norm for on-device LLMs rather than a nice-to-have.

Measure peak resident memory on your slowest target device, not your development machine. A model that fits in 300 megabytes on a laptop can balloon to 600 on a low-end phone because of allocator behavior and OS memory pressure. Test early on real hardware.

Latency targets

Interactive interfaces need first-token latency under roughly 200 milliseconds and steady throughput after that. Users tolerate a two-second wait for a batch result but abandon a chat that stalls. Test on the slowest device in your support matrix, with a cold start and a warm cache, and with the screen on battery. Thermal throttling on phones can cut throughput by half after a minute of sustained inference, which no benchmark captures.

Quantization and smaller context windows help latency more than a marginally faster model does. Cutting context from 4096 to 1024 tokens often halves time-to-first-token with no quality loss for short tasks.

Quality trade-offs

Public leaderboards measure general capability, which is not your capability. A model that scores well on MMLU may still fail your narrow task because the benchmark does not resemble your data. Build an evaluation set of 30 to 50 real examples from your domain, including the awkward cases, and measure accuracy, latency, and memory on that set before committing to a model. Re-run the eval when you change quantization or runtime, because the same weights behave differently under different inference engines.

Also test failure modes, not just success. What does the model do on empty input, on a question outside its domain, on adversarial phrasing? A model that fails gracefully is more useful in production than one that scores two points higher on a benchmark.

A practical shortlist

For classification, retrieval, and embeddings: MiniLM-L6-v2, DistilBERT, and BGE-small. They are fast, tiny, and well supported. For text generation under 500 million parameters: SmolLM2, Qwen2.5-0.5B, and TinyLlama, with the caveat that generation quality at this size is limited and tasks should be scoped accordingly. For vision: MobileNet and MobileViT. For speech-to-text: Whisper-tiny and Whisper-base. For text-to-speech: Piper and Kokoro.

Browse quantized variants on the Hugging Face model hub, and check the ONNX Runtime documentation for deployment guidance. When in doubt, start with the smallest model that plausibly solves the task, measure, and only scale up when the eval says you must.