Using Kafka Streams API for predictive budgeting

What this post added

This post details how Pinterest uses the Kafka Streams API to build a predictive budgeting system for their ads infrastructure. The system aims to reduce ad overdelivery by calculating 'inflight spend' (cost of ads insertions not yet charged). Key technical contributions include: 1. Designing a system to handle tens of thousands of events per second with sub-10-second end-to-end delay and 100% uptime. 2. Choosing Kafka Streams for its millisecond delay guarantee and lightweight nature compared to Spark and Flink. 3. Implementing a plan involving an 'inflight spend' service that consumes ad insertion events, aggregates spend using Kafka Streams with a 10-second window store per adgroup, and fans out updates to thousands of consumer machines. 4. Optimizing performance by switching from hopping windows to tumbling windows, resulting in an 18x throughput improvement. 5. Employing delta encoding for adgroup IDs and lookup table encoding for spend data to achieve a 4x reduction in message size.