Resilient AI Infrastructure | Harvey

This post details the technical implementation of Harvey's centralized model inference client library, designed to manage AI model performance reliably across millions of daily requests. It introduces a distributed key-value store, an in-house proxy service, and a custom-built model health tracker for scalable, fault-tolerant interaction with AI model inference servers. Key technical contributions include active load balancing and monitoring of LLM resources, a weighted selection process for healthy model deployments based on SLIs (latency, success rate), and a centralized quota & rate limiting system using a Redis-backed approximate sliding-window token bucket algorithm for handling bursty traffic. A thin proxy compatible with the OpenAI API spec is described for forwarding external requests and adding instrumentation. Granular observability with strict burn rate alerts and detailed token accounting via an in-house telemetry events pipeline are also highlighted.