Configuration Distribution System

4 posts · Jul 2013 – Oct 2026

What this trail covers

Cloudflare's configuration distribution system has evolved from a custom-built system (Quicksilver) to address limitations in scalability, performance, and reliability. Quicksilver v1.5 introduces a proxy/replica model to reduce disk space usage by caching hot keys on disk instead of replicating the full dataset everywhere. This design faces challenges with consistency due to asynchronous replication and is being addressed with Multiversion Concurrency Control (MVCC) to maintain sequential consistency. Workers KV Instant now leverages Quicksilver for instant global replication of key-value data, offering significantly faster reads and writes compared to classic Workers KV, with a pricing model optimized for read-heavy workloads.

All Cloudflare trails

Oct 2026Introducing Workers KV Instant — powered by QuicksilverIntroduces Workers KV Instant, a new mode for Workers KV that utilizes the Quicksilver key-value store for instant global replication. KV Instant offers 100x faster p99 reads (under 2ms) and sub-second write replication (250ms p99), compared to classic Workers KV's slower replication and higher read latency. It supports the same Workers KV API (get, put, list, delete) but with differences in metadata support and list operation pagination. Pricing is optimized for read-heavy workloads with significantly lower read costs and higher storage/Class A operation costs compared to classic Workers KV. KV Instant is currently in private beta.
Nov 2025Finding the grain of sand in a heap of SaltThis post details the development of an automated system to diagnose and resolve failures in Cloudflare's Salt configuration management system. It describes the architectural problem of job results living on Salt Masters, leading to tedious manual triage. The solution involves caching job results and correlating them with Git commits, external service failures, and ad hoc releases. This system provides self-service mechanisms for root cause analysis, reducing release delays and toil for SRE teams.
Jul 2025Quicksilver v2: evolution of a globally distributed key-value store (Part 2)This post details the evolution of Cloudflare's Quicksilver key-value store from V1 to V1.5 and finally to V2, addressing scalability challenges related to disk space and cache hit rates. V1 stored all data on every server. V1.5 introduced proxy mode with a cache and replica servers. V2 further refines this by dedicating a few servers with large disks as 'storage servers' holding full datasets (replicas), while all other servers run Quicksilver in cache-only proxy mode. A 'relay' mode was introduced to manage connections to storage servers and distribute cache misses within a data center. A reactive prefetching mechanism was implemented where cache misses are published and subscribed to by other proxies in the same data center to improve cache hit rates. For specific latency-sensitive instances like 'dnsv2', a data center-wide sharded cache was introduced before querying replicas in other data centers to further boost cache hit rates.
Jul 2025Quicksilver v2: evolution of a globally distributed key-value store (Part 1)This post details the evolution of Cloudflare's Quicksilver key-value store from v1 to v1.5. Quicksilver v1.5 introduces a proxy/replica role split to address disk space inefficiencies caused by replicating the full dataset globally. Proxies act as persistent caches (using RocksDB) for hot keys, while replicas store the full dataset. This design aims to reduce disk space by up to 50% based on analysis showing working sets are typically 1-20% of the keyspace. The post also introduces the consistency challenges posed by asynchronous replication in this new architecture and outlines the use of Multiversion Concurrency Control (MVCC) with a new RocksDB column family to maintain sequential consistency by tracking multiple versions of keys.
Nov 2020Moving Quicksilver into productionThis post details the migration of Cloudflare's core distributed data store from Kyoto Tycoon (KT) to Quicksilver. It describes the challenges of deploying a new system in a fault-tolerant production environment without downtime. Key technical contributions include: the development of QSKTBridge for replicating data from KT to Quicksilver, the use of Compare And Swap for preventing duplicate entries during replication, the implementation of memcached and KT HTTP protocols for transparent client access, the use of LMDB for crashproof storage and its bootstrap mechanism, the identification and mitigation of replication saturation caused by I/O contention during bootstrapping, the validation of Quicksilver's performance through FL metrics showing improved error rates and response times, the discovery of I/O errors and write amplification from aging SSDs and LMDB's copy-on-write mechanism, the implementation of a workaround for replication delays caused by saturated I/O, the removal of a cascading disconnect feature that caused replication lag, and the identification of a need for dynamic topology configuration using DNS.
Advertisement
Mar 2020Introducing Quicksilver: Configuration Distribution at Internet ScaleThis post details the challenges encountered with using Kyoto Tycoon (KT) for configuration distribution at Cloudflare's scale, specifically highlighting issues with exclusive write locks degrading read latency and data corruption due to disabled fsync. It introduces the need for a new system and outlines the architectural requirements for a scalable, reliable configuration distribution mechanism.
Jul 2015Blue Light Special: Ensuring fast global configuration changesThis post details the development of a custom alert system using a Raspberry Pi and a police light to visually monitor global configuration propagation times for DNS records. It highlights the critical need for near-instantaneous configuration changes across Cloudflare's global network and the challenges associated with ensuring rapid propagation of customer configurations (DNS records, WAF rules, etc.). The system monitors propagation from external machines and triggers a visual alert when thresholds are exceeded.
Jul 2013Integrating Kyoto Tycoon With PostgreSQLIntroduced a Foreign Data Wrapper (FDW) for Kyoto Tycoon (KT) in PostgreSQL 9.3. This allows KT data to be represented as a database table within PostgreSQL, enabling the use of database triggers for data synchronization. This reduces latency and simplifies synchronization logic by translating PostgreSQL transactions into KT transactions, exporting ACID guarantees to KT.