Blogs›Apple›On-device Audio Synthesis

On-device Audio Synthesis

posts

2026

This post introduces a memory-efficient audio synthesis architecture for on-device deployment, enabling real-time, high-fidelity speech generation within strict compute and memory constraints. The architecture decouples temporal and depth processing using a residual vector quantization (RVQ) representation with a streaming encoder, a temporal decoder, and a depth decoder. A single, reusable depth decoder with Diffusion Transformer (DiT)-style stage conditioning generates RVQ levels autoregressively. Causal sliding window attention with fixed-window key-value caching ensures constant memory complexity. Deployed on the Apple Matrix Coprocessor (AMX), it achieves ~10ms per generation step with low memory footprint and asset size, supporting long audio streams. This system is deployed in Siri Expressive Voices, improving Mean Opinion Score (MOS) for speech synthesis.

2026

Memory Efficient Audio Synthesis with Decoupled Temporal Depth Diffusion Transformers

7/28/2026

Introduces a novel memory-efficient audio synthesis architecture for on-device deployment, featuring a decoupled temporal and depth processing design using RVQ. Key innovations include a single reusable depth decoder with DiT-style conditioning and causal sliding window attention with fixed-window key-value caching for constant memory complexity. This architecture enables real-time synthesis within tight resource constraints and has been deployed in Siri Expressive Voices, demonstrating significant improvements in audio quality metrics.