Kubernetes Control Loops

5 posts · Jul 2022 – Jun 2026

What this trail covers

PlanetScale leverages Kubernetes to manage database clusters, extending its control loop capabilities with a Vitess Operator for stateful workloads. This automates deployment, storage management, backup processes, and failure mitigation, ensuring high availability and resilience. The system handles pod/container failures and AZ outages, with databases distributed across multiple availability zones for fault tolerance. Replicas are also stored in separate AZs, and automatic failover to a replica. This post details the fundamental concepts of control loops and how they are implemented in Kubernetes operators for managing stateful systems like databases.

All PlanetScale trails

Jun 2026The feedback loops behind KubernetesThis post introduces the concept of control loops and feedback controllers, explaining how they are fundamental to Kubernetes' ability to manage workloads at scale. It details how to manually manage a PostgreSQL database, highlighting the challenges of ephemeral storage, node selection, disk management, high availability, and inter-instance communication. The post then maps these manual steps to the components of a closed feedback loop (setpoint, measured output, error, controller, actuator, plant) and explains how Kubernetes provides the necessary primitives (store, watches, queues, retries, leader election) to build robust operators for stateful systems.
Sep 2023Scaling hundreds of thousands of database clusters on KubernetesThis post details how PlanetScale uses Kubernetes and a Vitess Operator to manage a large fleet of database clusters. It explains the challenges of running stateful databases on Kubernetes, contrasting PlanetScale's approach of using the Vitess Operator with StatefulSets for storage management. It also describes their custom orchestration for new database deployments, a specialized backup strategy using dedicated tablets, and multi-AZ provisioning for high availability.
Aug 2023What makes up a PlanetScale Vitess database?This post details the core components of a PlanetScale Vitess database, explaining how Vitess and Kubernetes work together. It describes Vitess components like Tablets, vtgate, and vtctld, and how they are deployed as pods within Kubernetes. The post clarifies that each PlanetScale database and branch is an independent Vitess cluster. It also introduces PlanetScale's edge connectivity infrastructure for improved database access, online schema change capabilities leveraging Vitess, database branching and deploy requests with safety checks, schema revert functionality using shadow tables, automated and validated backups, the provision of primary and replica instances across availability zones with automated failover, and built-in query performance monitoring with Insights.
May 2023Pulling back the curtain: the new database overview pageThis post introduces a new database overview page in the PlanetScale dashboard that visualizes the underlying infrastructure of a database. It details the display of load balancers, primary databases, replicas, read-only regions, and sharded database components (keyspaces and shards). For each component, it outlines the metrics that are now visible, such as connections, latency, QPS, reads/writes per second, IOPs, CPU, and memory. It also highlights the display of replication lag for replicas and the ability to drill down into individual shard details. This enhances the observability of the PlanetScale platform's architecture.
Jul 2022Consensus algorithms at scale: Part 8 - Closing thoughtsThis post concludes a series on consensus algorithms by reconceptualizing key aspects: pluggable durability, revocation/establishment of leadership, handling races (favoring lock-based approaches for flexibility), and completing/propagating requests with versioning. It details how Vitess implements these concepts, specifically mentioning vtorc's plugin API for durability and the Vitess Operator for graceful failover, aiming for full auto-pilot operation. Future topics like failure detection, consistent reads, and node addition/removal are noted as separate concerns.
Advertisement