Compressing Streaming Neural Audio Encoders via Latent-Space Distillation

This post details a latent-space distillation technique for compressing streaming neural audio encoders. The method trains a student encoder to regress the pre-quantizer latent representation of a teacher encoder, using a squared-error objective. A single affine layer handles width mismatches. This approach achieves up to 2.8x compression with minimal Word Error Rate (WER) degradation.