The Query Strikes Again

This post details a database incident caused by a large user deletion job that overloaded a Vitess shard, leading to replication lag, OOM kills of MySQL processes, and an infinite loop of replica promotion and replacement failures. The incident was resolved by manually provisioning larger instance types for replicas and temporarily disabling automation for host replacement. Additionally, the 'forget user' job was optimized by fixing inefficiencies in how it processed channel subscriptions and updated thread subscriptions, reducing the number of database queries and the overall workload.