How we made one of our largest inference workloads 4.7× more GPU-efficient | Decagon

This post details optimizations for a scheduled inference workload, achieving a 4.7x improvement in fleet-level GPU efficiency. Key contributions include disaggregating prompt processing from token generation to allow for specialized batching and state transfer, implementing an admission controller to manage traffic bursts and ensure redistributability of requests, and reducing worker startup time by prefetching model files. These changes address bottlenecks in batch formation, memory pressure, state transfer, burst admission, and worker startup.