Load Balancing AI Services for Availability and Speed

What this post added

Introduced a service-aware load balancer for Pinecone Assistant using the 'power of two choices' algorithm. Implemented distinct scoring policies for embeddings/rerankers (latency+reliability) and LLMs (reliability+load). Demonstrated improved latency and availability during upstream incidents through staged rollouts.