How Meta animates AI-generated images at scale

This post details optimizations for Meta AI's image animation feature, focusing on reducing inference latency and improving scalability. Key technical contributions include: halving floating-point precision from float32 to float16 (bfloat16) to reduce memory footprint and speed up operations; optimizing temporal-attention expansion by performing it after cross-attention's linear projection layers to reduce compute and memory; leveraging DPM-Solver with a linear-in-log signal-to-noise time to reduce sampling steps to 15; combining guidance and step distillation to reduce three forward passes per step to one, and distilling 32 teacher steps into eight student steps with one forward pass per step; and applying PyTorch optimizations like TorchScripting (continuous folding, fusing operations, reducing computational graph complexity) and freezing, later migrating to PyTorch 2.0 for more granular component optimization and multi-GPU inference. For large-scale deployment, a regional traffic management system was implemented to keep requests in the same region as their requester, minimizing network overhead. This system dynamically calculates routing tables based on load data, predefined thresholds, and routing rings to offload traffic to nearby regions when nearing capacity. To address dips in success rate due to GPUs being saturated, a probing system approximating retries was used to quickly check for free GPUs and prevent failures, with optimized retry settings for the router.