TechPrufer
Get started
PlanetScale
PlanetScale logoPlanetScale · Feature trail

Database Traffic Control

14 posts · Jul 2022 – Oct 2026

What this trail covers

PlanetScale offers enhanced database traffic control by allowing users to attach resource budgets to specific slices of Postgres traffic, enabling isolation of critical flows from resource-intensive operations. This feature also provides mechanisms for managing background jobs and routing connections to the nearest read replica globally through a dedicated PlanetScale Global Network. This network acts like a CDN for database connectivity, terminating connections at the edge, pooling them, and ro

All PlanetScale trails
  1. What today's software owes to Bronze Age clocksThis post details the implementation and configuration of PlanetScale's Database Traffic Control feature, which acts as an admission control system for Postgres. It explains the historical context of leaky bucket algorithms and their application in software, particularly for rate limiting and queuing. The post elaborates on the three main settings for resource budgets: server share and burst limit (token bucket), per-query limit, and maximum concurrent workers. It highlights how these settings allow for granular control over database traffic to prevent overload and ensure system health, and emphasizes the importance of reasoning about expected failure states when implementing such controls.
  2. Concurrency vs. Throughput: why more parallelism can make databases slowerThis post details a production incident where a long-running transaction and excessive concurrent requests led to a database meltdown. It explains how migrating from a thread-pool-based connection manager (Cloud SQL) to a transaction-pool-based one (Vitess) exacerbated the issue. The core problem was identified as retrograde scaling due to high concurrency, where increased in-flight requests led to mutual slowdowns (coherency cost) rather than increased throughput. The solution involved reducing the Vitess transaction pool size and implementing queuing with longer timeouts, effectively mimicking the behavior of the previous thread pool. This configuration successfully handled a subsequent high-traffic burst without errors or throughput degradation. The post also discusses the applicability of these principles to workloads with pessimistic locking and shared state, and mentions similar solutions like PgBouncer for PostgreSQL.
  3. When the Postgres query planner goes rogueThis post details a specific incident where an unexpected PostgreSQL query planner behavior caused a database CPU spike to 100%. It demonstrates how PlanetScale's Database Traffic Control feature was used to immediately apply a resource budget to the problematic query pattern, preventing its execution and restoring database health. The post explains the underlying cause of such planner misbehavior (stale statistics) and outlines the steps for a long-term fix (using EXPLAIN ANALYZE, refreshing statistics, adjusting indexes, or rewriting queries). It reinforces the utility of Traffic Control for immediate mitigation of performance incidents.
  4. Deadlocks and downtimeThis post details how PlanetScale's Traffic Control feature can be used to mitigate deadlocks in Postgres databases. It explains that deadlocks occur when transactions acquire locks in conflicting orders, leading to blocking and potential downtime. The post highlights how Traffic Control, by allowing resource budgets to be attached to queries (specifically via query tags), can limit the concurrency of problematic queries. By setting a small capacity of concurrent workers for queries identified as causing deadlocks, Traffic Control can reject matching queries before they acquire too many locks, thus reducing the likelihood of deadlock formation. The post also reinforces the importance of application-level retry logic with backoff and jitter as a complementary strategy for handling deadlocks.
  5. Patterns for Postgres Traffic ControlThis post provides practical Go code examples for implementing PlanetScale's Traffic Control feature. It details how to use SQLCommenter for tagging queries with context like service roles, HTTP routes, deployment tags, and subscription tiers. It also shows how to manage background jobs and scripts by setting distinct `application_name` values in connection strings. The post elaborates on the patterns for isolating traffic, managing risk during deployments, and enforcing tier-based limits.
  7. Graceful degradation in PostgresThis post details how to categorize Postgres traffic by priority (critical, important, best-effort) and tag queries using sqlcommenter. It explains how to set up resource budgets in PlanetScale's Traffic Control feature, including configuring server shares, burst limits, and per-query maximums. The post emphasizes starting budgets in 'warn' mode for tuning before enforcing, and describes how to dynamically disable traffic during crisis events to prevent total database outages.
  8. Enhanced tagging in Postgres Query InsightsThis post enhances the Query Insights feature by integrating query tags into aggregated data, enabling new capabilities such as viewing tag distribution per query pattern, searching queries by tag, and seeing per-tag breakdowns of database-level statistics. The implementation details the shift from attaching tags only to individual query logs to including them in aggregate summaries. It explains two methods for sending tags: a simple accumulation that loses attribution, and a more robust approach of emitting separate aggregate messages per tag combination. To manage the potential explosion of messages from high-cardinality tags, two cardinality reduction strategies are detailed: per-tag limits (collapsing tags per query pattern if unique values exceed a threshold) and per-interval limits (collapsing tags within a 15-second interval if the number of tag combinations exceeds a threshold). The post also covers tracking when tag collapsing occurs.
  9. Behind the scenes: How Database Traffic Control worksThis post details the internal implementation of Database Traffic Control, explaining how it leverages Postgres extensions and hooks (ExecutorRun, ProcessUtility) to monitor query execution. It describes the four checks performed before query execution: rule matching, concurrency limit check, per-query cost limit check, and available capacity check. The post elaborates on cost prediction using planner cost estimates and exponential moving averages to derive a 'k' constant for estimating CPU/wall-clock time. It also explains the leaky-bucket rate limiting model for budgets, including burst limits and server shares, and the inverted model for debt management and eviction. Finally, it introduces the rule set data structure for efficient rule evaluation and the strategy for managing buckets in shared memory.
  10. Introducing Database Traffic ControlIntroduces 'Database Traffic Control' for Postgres, a system that allows users to define flexible resource budgets for subsets of database traffic. Budgets can be defined based on query patterns, application names, Postgres users, or custom tags, and can enforce limits on CPU usage, burst limits, backend process concurrency, and per-query timing. The system supports 'warn' and 'enforce' modes, and can be configured via the dashboard, API, or CLI. It also highlights the use of SQL comments for tagging traffic.
  11. Anatomy of a Throttler, part 3This post, the third in a series on throttlers, delves into the client-side aspects of throttling. It introduces the concept of client identification for better analysis and operational control, discussing the risks of exemption and starvation. It proposes prioritization through de-prioritization using a probabilistic approach and explores throttling based on different metrics, highlighting the nuances of exemption. The post also covers categorization and breakdown of client identities for granular control, the importance of time-limited rules, and contrasts cooperative throttling with enforcement models like proxies.
  12. Anatomy of a Throttler, part 2This post explores singular and distributed throttler designs, discussing trade-offs in high availability, metrics access, and scalability. It introduces the Vitess tablet throttler as a case study for granular throttling scopes and discusses strategies for reducing the throttler's impact, such as metric hibernation and adjusting collection granularity. The post highlights the challenges of synchronicity and staleness in distributed systems.
  13. Anatomy of a Throttler, part 1This post introduces the concept of a throttler for database systems, focusing on asynchronous, batch, and massive operations. It discusses various metrics for determining when to throttle, including replication lag, threads_running, queue delay, queue length, load average, and pool usage. The post emphasizes the importance of using multiple metrics and understanding their limitations as symptoms rather than direct causes. It also touches on how introducing a throttler can reveal existing synchronization issues in applications.
  14. Introducing global replica credentialsThis post introduces global replica credentials and the underlying PlanetScale Global Network architecture that enables them. It details how these credentials automatically route reads to the nearest read replica globally, leveraging a CDN-like connectivity layer. The post explains the components of a connection (Route, Credential, Endpoint) and how the PlanetScale Global Network terminates connections at the edge, performs connection pooling, and routes traffic to the origin database. It highlights the dynamic nature of Routes and the use of latency measurements for real-time routing decisions, allowing seamless addition and removal of read-only regions without client-side changes.
  15. $ pscale pingIntroduced `pscale ping` CLI command to measure network latency from the user's location to various PlanetScale database regions, aiding in optimal region selection for development or application deployment.
  16. Introducing IP restrictionsIntroduces IP restrictions as a beta feature for database branches, allowing admins to define allowed IP address ranges for password connections. This adds an extra layer of security beyond default password access controls, useful for segmenting access, enforcing bastion usage, limiting single-client access, and meeting compliance requirements.
  17. Faster MySQL with HTTP/3Introduces an experimental HTTP/3-based API for MySQL, comparing its performance against the traditional MySQL binary protocol. The experimental API uses `connect-go` and QUIC for HTTP/3 transport. Tests were conducted across high-latency and low-latency network conditions, evaluating connect+SELECT 1, parallel SELECT 1, medium/large SELECTs, and medium/large INSERTs. Results show significant improvements in cold start times for HTTP/2 and HTTP/3 over the MySQL protocol, attributed to 0-RTT TLS handshakes. Larger payloads also benefit from HTTP/2 and especially HTTP/3 due to compression and QUIC's handling of network issues, though some performance regressions were noted in low-latency scenarios with HTTP/3, potentially due to immature QUIC implementations.
  18. Behind the scenes: How we built Password RolesThis post details the implementation of granular password roles for PlanetScale databases. It explains how Vitess's static ACL configuration was adapted for a dynamic, customer-facing system. The solution involves storing user credentials and their mapped roles in an external database, which is then used to rewrite the security principal on-the-fly. This allows for immediate application of role changes and ensures consistent authorization state across all database pods, overcoming limitations of Vitess's static ACL file reloading mechanism.