Offline Voice Control
On-device voice control for macOS
A voice command system for macOS whose speech never leaves the machine: wake word, transcription, speaker verification and intent resolution all run locally, inside a latency budget on a laptop CPU. It will open a web search or reach a remote box over SSH when you ask it to. What it will not do is send your voice anywhere.
The problem
Voice control on a laptop is either useless or a privacy problem. The useful assistants stream audio to a datacentre, which rules them out for anyone handling client work under an NDA, and the offline options recognise a fixed dozen phrases. The interesting constraint is not accuracy in the abstract — it is accuracy within a latency budget on a CPU that is also compiling something.
Our approach
A staged pipeline where each stage can be swapped and measured independently: voice activity detection gates a local faster-whisper transcription, a speaker encoder exported to ONNX checks the voice against an enrolled print, and an intent resolver scores the transcript against an editable command grammar. Every stage records its inputs, outputs and timing, so a wrong action can be explained rather than guessed at. Actions reach macOS through osascript and the Accessibility API rather than synthetic keystrokes, so they stay inspectable. Torch is deliberately not a runtime dependency — it and CTranslate2 each bundle their own OpenMP and abort when loaded together, which is why the encoder runs on onnxruntime instead.
The outcome
A tool that runs entirely on the machine it controls, where a misfired command produces a readable trace instead of a shrug — and where adding a command is a grammar edit, not a code change.
What is inside
Wake word ("nova"), transcription and intent resolution all on-device
No audio leaves the machine: transcription runs on faster-whisper locally, and speaker verification on an exported ONNX encoder
Speaker verification against an enrolled voiceprint, so it answers to one person
Intent resolver scoring transcripts against an editable command grammar
Per-stage timing and input/output capture, so a wrong action is explainable
System actions through osascript and the Accessibility API rather than synthetic keystrokes
Command grammar editable at runtime without a restart
Typed command palette fallback when the audio device is unavailable
Nothing to configure by hand: apps are scanned from the application folders, settings panes come from Apple’s URL scheme, and files are found through Spotlight
SSH credentials for remote machines live in the macOS Keychain, never in the config file
Engineering notes · 5 min read
The constraint is the product
0
Network calls
by design, not by default4
Pipeline stages
each independently measuredGrammar edit
Adding a command
not a code changeAnyone can build voice control by posting audio to a hosted speech API. That is also exactly what makes it unusable for someone under an NDA: the interesting sentences are the ones you cannot send anywhere.
Running everything locally turns a product decision into an engineering one. Wake-word detection, transcription and intent resolution all have to fit inside a latency budget on a CPU that is frequently already busy compiling something. The budget, not the accuracy ceiling, sets the design.
Four stages, each replaceable
| Stage | Cost | Why it is its own stage |
|---|---|---|
| Wake word | Cheap, always on | Gates everything expensive downstream |
| Voice activity | Very cheap | Trims silence before transcription pays for it |
| Transcription | Expensive, bursty | Model can be swapped for a smaller one on battery |
| Intent resolution | Cheap | Pure function of transcript plus grammar — trivially testable |
Every stage records its inputs, outputs and timing. That trace is the difference between a tool you trust and one you stop using: when the wrong thing happens, you can see whether the transcript was wrong or the intent scoring was, and fix the right one.
Actions should be inspectable
- System actions go through PyObjC and AppleScript, not synthetic keystrokes
- The command grammar is data, editable at runtime with no restart
- History lives in local SQLite with no remote sync, ever
- A typed command palette takes over when no audio device is available
- Failed resolutions are kept with their scores, so the grammar can be improved from evidence
Synthetic keystrokes work until they land in the wrong window. A scripted action either targets the right application or fails loudly.
The demo on this page uses typed input rather than a microphone. That is not a shortcut: a static demo must not request device permissions, and the thing worth showing is the intent-resolution trace, which is identical whether the transcript arrived from Whisper or from a keyboard.
Core
Audio
Speaker ID
macOS integration
Secrets
Platforms
macOS
Need a platform like Offline Voice Control?
We can walk you through how this was built, what it cost to run, and what the equivalent looks like for your business.
Or email us directly at [email protected]