Memory Efficient Audio Synthesis with Decoupled Temporal Depth Diffusion Transformers

Introduces a memory-efficient audio synthesis architecture for on-device deployment, featuring a detokenizer that converts semantic audio tokens into high-fidelity audio. The architecture decouples temporal and depth processing using a residual vector quantization (RVQ) representation with a streaming encoder, temporal decoder, and depth decoder. A single reusable depth decoder with Diffusion Transformer (DiT)-style stage conditioning generates RVQ levels autoregressively. Causal sliding window attention with fixed-window key-value caching provides constant memory complexity. Deployed on the Apple Matrix Coprocessor (AMX), it achieves ~10ms per generation step, ~21MB peak runtime memory, and ~329MB on-device assets, enabling continuous streaming synthesis. This approach replaces linear and quadratic memory scaling of prior methods and improves MOS scores.