Storing time series data in sharded MySQL to power Query Insights

This post details the implementation of a scalable telemetry pipeline for PlanetScale's Query Insights tool. It describes how time series data, specifically query pattern-level aggregations and slow query events, is collected from VTGate instrumentation, sent to Kafka, and then processed by consumers to be stored in a sharded PlanetScale (MySQL/Vitess) database. Key technical contributions include: implementing instrumentation in Vitess to send aggregate summaries and slow query events to Kafka; using Kafka keys for deterministic sharding of data by database ID and query fingerprint; employing in-memory coalescing in Kafka consumers to reduce database writes; sharding the Insights database on customer database ID for scalability; using DDSketch for error-bounded quantile estimation of query latencies; and developing custom C++ MySQL functions to read/write DDSketch binary format and compute percentiles directly within MySQL.