On-device Audio Synthesis

Siri Expressive Voices synthesizes rich, configurable speech in real time and entirely on device, powered by AFM 3 Core Advanced. This work presents the memory-efficient audio synthesis architecture: a detokenizer that converts semantic audio tokens into high-fidelity audio within the tight compute and memory budget of the Apple Matrix Coprocessor (AMX). The architecture decouples temporal and depth processing using a residual vector quantization (RVQ) representation with a three-component design—a streaming encoder, a temporal decoder, and a depth decoder. A single reusable depth decoder with Diffusion Transformer (DiT)-style stage conditioning generates all RVQ levels autoregressively, replacing dedicated per-level decoders. Causal sliding window attention with fixed-window key-value caching yields constant memory complexity. Deployed on the AMX, the detokenizer sustains roughly 10ms per generation step with a peak runtime memory of only ~21MB, enabling continuous streaming synthesis. This constant, small footprint replaces the linear and quadratic memory scaling of conventional transformer- and GAN-based approaches. Audio quality assessment confirms that the proposed architecture maintains synthesis fidelity while achieving computational efficiency gains. The proposed architecture is deployed in production as part of Siri Expressive Voices, powering a voice overhaul with Pace and Expressivity customizations sliders in Apple Devices and support for custom assistant voices. Operating at a 1-billion-parameter activation size within AFM 3 Core Advanced, it improves Mean Opinion Score (MOS) by +0.28 overall (4.15 vs. 3.87) and by +0.42 on conversational speech (4.24 vs. 3.82) over the prior on-device text-to-speech system.