Scaling real-time AI agents with session-aware load balancing- Google Developers Blog

This post introduces the challenges of scaling real-time AI agents, which rely on stateful, long-lived bidirectional streams rather than ephemeral requests. It highlights the inadequacy of traditional QPS and CPU utilization metrics for load balancing these systems. The post proposes tracking active sessions within the runtime as a critical signal for load balancing. It details a hybrid load balancing model that combines session counts with utilization metrics, using a formula to estimate effective capacity and prevent backends from becoming overloaded. The post also emphasizes the importance of realistic benchmarking for session-aware balancing and validating the overhead of session tracking mechanisms.