Vol. XXIV · Making Inference Cheap Plate 9 of 10on-device, edge inference, phones, laptops, memory bandwidth, quantisation, Apple Intelligence, unified memoryCE

Models on Devices.

One user, a few gigabytes, tens of gigabytes per second. Divide.

A device serves one person, so there is no batch to fill: every token is a single-user decode step, bounded by how fast memory delivers the weights (plate XXIII.1). A laptop’s unified memory delivers 120 to 546 GB/s across Apple’s M4 family; a phone less; an H100 3 350. The model must also fit in memory shared with the operating system and apps, and the energy per token comes from a battery.

Apple’s published description of its on-device model shows the resulting design: a few billion parameters, compressed to between three and four bits per weight on average by mixing 2- and 4-bit palettes, generating tens of tokens per second on an iPhone 15 Pro. The workshop computes the bandwidth bound for a model of the chosen size and precision on the chosen device.

Below: the single-user decode bound for each model size on the chosen device at the chosen precision; readouts give the weight bytes and bound for the chosen size.

What you will learn
Workshop
Bandwidth: Apple M4 120 GB/s, M4 Pro 273 GB/s, M4 Max 546 GB/s (Apple specifications); H100 3.35 TB/s. Bound: 80% of peak bandwidth over weight bytes; cache, compute, other apps and thermal limits ignored. Apple’s on-device model figures from Apple’s June 2024 description.

Bandwidth is the budget

Apple’s on-device model, about 3 billion parameters at 3.7 bits per weight, occupies about 1.39 GB; generating 30 tokens a second means reading it 30 times a second, at least 42 GB/s of memory traffic — which tells you the phone’s memory system must supply that much, and why a larger model would not fit the budget. Prompt processing, at about 0.6 ms per token, is compute-bound and much faster per token, as on a server.

On laptops the same division sets what is practical: a 3-billion-parameter model at 4 bits decodes at up to about 64 tokens a second on a base M4; an 8-billion model in 16 bits at about 27 on an M4 Max; a 70-billion model at 4 bits (35 GB) fits in an M4 Max’s up to 128 GB of memory but decodes at about 12. Every technique in this volume — quantisation above all, then distillation, pruning and speculation — exists in its most aggressive form on devices, where the budget of bytes per token is smallest.

When a model does not fit in fast memory, weights can be streamed from flash storage as needed, at much lower bandwidth — the subject of Apple’s “LLM in a Flash” work.

Terms

On-device inference
Running the model on the user’s own phone or computer.
Unified memory
One memory pool shared by CPU and GPU, as in Apple silicon.
Palettization
Storing each weight as an index into a small table of values.
Bandwidth budget
Bytes per second available to read weights.
On a device, tokens per second is bandwidth over model bytes; that budget explains small, aggressively quantised on-device models.

Sources

  1. Apple Machine Learning Research, “Introducing Apple’s On-Device and Server Foundation Models”, June 2024.
  2. Apple, M4 Pro and M4 Max announcement, 30 October 2024; Apple M4 specifications.
  3. Alizadeh et al., “LLM in a Flash: Efficient Large Language Model Inference with Limited Memory”, ACL 2024.