TechPrufer
Get started
PlanetScale
PlanetScale logoPlanetScale · Feature trail

Incident Response and Resilience

5 posts · Jun 2020 – Nov 2025

What this trail covers

PlanetScale leverages Kubernetes to manage database clusters, extending its control loop capabilities with a Vitess Operator for stateful workloads. This automates deployment, storage management, backup processes, and failure mitigation, ensuring high availability and resilience. The system handles pod/container failures and AZ outages, with databases distributed across multiple availability zones for fault tolerance. Replicas are also stored in separate AZs, and automatic failover to a replica. This post details the principles of extreme fault tolerance, including isolation, redundancy, and static stability, applied to their architecture (control and data planes, database clusters) and processes (always failing over, synchronous replication, progressive delivery) to mitigate non-query-path, cloud provider (instance, zonal, regional), and PlanetScale-induced failures.

All PlanetScale trails
  1. Report on our investigation of the 2025-10-20 incident in AWS us-east-1This post details an incident in AWS us-east-1 caused by DNS misconfiguration and subsequent capacity constraints and network instability. It describes two phases: the first impacting the PlanetScale control plane due to upstream AWS service outages (DynamoDB, S3, STS), and the second impacting some database branches due to EC2 instance launch failures and network partitions. The post highlights PlanetScale's resilience through control/data plane isolation and automated failover, but also identifies areas for improvement, including reducing dependencies on SaaS providers and better utilizing AWS availability zones to mitigate network partition failures.
  2. The principles of extreme fault toleranceThis post details the principles of extreme fault tolerance (isolation, redundancy, static stability) and how they are applied to PlanetScale's architecture (control plane, data plane, database clusters) and processes (always failing over, synchronous replication, progressive delivery). It elaborates on specific failure modes and mitigation strategies for instance, zonal, and regional cloud provider failures, as well as PlanetScale-induced failures, emphasizing the role of feature flags and gradual rollouts.
  3. Is your database bleeding money?This post details the financial and operational costs associated with database downtime, manual database management, and complex database DevOps processes. It quantifies these costs with real-world examples and highlights how these issues lead to lost revenue, decreased user trust, and reduced engineering velocity. The post then presents PlanetScale's managed Vitess platform as a solution, emphasizing its features like Git-like workflows for schema changes, automated backups, no-downtime version updates, horizontal sharding, and safe migration capabilities (branching, revert feature) as direct mitigations for these costly problems.
  4. We now display PlanetScale system status directly in your dashboardThis post details the implementation of displaying PlanetScale system status directly in the dashboard. It describes using Vercel Edge Functions to proxy and cache the status page API, achieving ~30ms response times. The edge function code is provided, along with the `cache-control` header configuration for optimal caching. A React component using `useSWR` to fetch and display incident data from the edge function is also included.
  5. Debunking 3 myths about Vitess fault toleranceThis post debunks three myths about Vitess fault tolerance: 1) that it loses data without a consensus-based commit protocol (explaining the use of MySQL's lossless semi-synchronous replication), 2) that self-recovery is impossible upon master failure (highlighting Orchestrator and PlanetScale's Vitess operator for automatic reparenting), and 3) that configuring self-recovery is complex (detailing Vitess's built-in PlannedReparentShard and EmergencyReparentShard workflows).