How we scaled the GitHub API with a sharded, replicated rate limiter in Redis

This post details the migration of GitHub's rate limiter from Memcached to a sharded, replicated Redis backend. It describes the challenges with the old system (Memcached persistence, datacenter routing) and the proposed solution using Redis with client-side sharding, Lua scripts for atomicity, and Redis expiration. It also details two critical bugs encountered post-release: 'reset at' header wobbling due to clock drift between Redis and Ruby, and incorrect rate limit rejections caused by stale data read from Redis replicas. Fixes involved managing 'reset at' in application code and ensuring responses accounted for replica reads and potential expirations.