ZGateway: Learnings from Putting a Proxy in Front of ZippyDB

This post introduces ZGateway, a new proxy layer designed to manage traffic for ZippyDB, Meta's widely-used key-value store. ZGateway addresses the challenges of a sprawling ZippyDB client fleet by acting as a unified entry point. Key technical contributions include: 1. **Connection Management:** ZGateway collapses the many-to-many mesh of TLS connections between clients and database hosts into bounded connections at the proxy tier, significantly reducing resource consumption and improving reliability by containing reconnection storms. 2. **Request Batching and Coalescing:** ZGateway groups requests from multiple clients destined for the same shard into single backend RPCs, amortizing fixed overheads, reducing QPS, and smoothing load. It also coalesces requests for hot keys, preventing stampedes against replicas. 3. **Traffic Routing and Migration:** ZGateway enables incremental and reversible migration of traffic via client-side configuration flags, allowing for controlled rollouts and rollbacks. 4. **Tenant Isolation and Admission Control:** The proxy tier enforces per-tenant admission control, validation, and shaping, preventing misbehaving tenants from impacting others. 5. **Read-Through Cache:** A caching tier is integrated to serve read requests directly, further improving performance. The post details the arithmetic behind fan-in/fan-out reduction and the safety mechanisms for in-memory batching (idle eviction, in-flight caps).