How on device dictation actually works on Apple silicon
Why a speech model that used to need a data centre now runs on a laptop, what the Neural Engine has to do with it, and where the remaining limits are.
Ten years ago, dictating to a computer meant sending your voice to a data centre. That was not a business decision, it was arithmetic: the models were too large and too slow for the hardware on a desk. The reason a wave of local dictation apps appeared at roughly the same time is that the arithmetic changed.
What changed
Three things, roughly in order of importance.
Models got smaller for the same accuracy. A speech model that once needed billions of parameters to be usable now gets there with far fewer, because of better training rather than better hardware.
Apple silicon put a Neural Engine on every Mac. It is a block of the chip built for the specific kind of arithmetic these models are made of, and it does that work at a fraction of the energy the CPU would need. A laptop can run it continuously without the fan coming on or the battery noticeably suffering, which is what makes an always available dictation key realistic rather than a demo.
Unified memory removed the copy. On these chips the CPU, the GPU and the Neural Engine address the same memory, so audio does not have to be shuffled between separate pools on its way through. For something latency sensitive, avoiding copies matters more than it sounds.
Where the time actually goes
The interesting number in dictation is not throughput, it is the gap between you finishing a sentence and the sentence appearing. On a cloud service most of that gap is not computation at all: it is the round trip and whatever queue is in front of you. That floor is set by the network, and no amount of local hardware lowers it.
Locally there is no trip. Zumbo composes a sentence in about a fifth of a second, which is under the threshold where people stop trusting what they said and start waiting to see what appears. It is the same figure with the network off, because the network was never in the path.
What this does not fix
Being precise about the limits, since this is an engineering post.
It needs the hardware. Apple silicon and macOS 14 or later, because the Neural Engine is doing the work. On an Intel Mac this whole approach is unavailable and Apple’s own dictation, which falls back to its servers on older machines, is the right answer.
It does not make a general model know your words. Running locally changes where the computation happens, not what the model has seen. A library name or a drug name is still not in ordinary English, which is why vocabulary packs and a correction that sticks exist alongside the engine rather than as a substitute for it.
And it costs download size. The engine ships inside the app, which is why Zumbo is about 570 MB. That is the honest price of not having a server, and the reasoning is here.
More in why 200 ms matters and how Zumbo keeps your voice on your Mac.
Hold it to talk, tap it to toggle, Escape cancels. Pick the key in Settings.
Free for 3 days, everything unlocked. Then $15 once. macOS 14 or later, Apple silicon.