Meta’s Full-stack HHVM optimizations for GenAI

This post details full-stack HHVM optimizations for GenAI workloads. Key contributions include splitting GenAI inference traffic into a dedicated WWW tenant for specialized runtime and warm-up configurations, leading to a 30% latency improvement. Specific optimizations include increasing the request runtime limit, adjusting thread-pool sizing for longer-running requests, leveraging HHVM's Jump-Start for faster JIT compilation, implementing request warm-up to pre-cache configuration and service discovery info, and using request shadowing to maintain JIT cache coverage for feature rollouts.