Using Kafka Streams API for predictive budgeting

This post details how Pinterest built a predictive budgeting system using the Kafka Streams API to prevent ad overdelivery. It explains the problem of overdelivery due to real-time spend data latency and the need for predictive spend. The system requirements included handling tens of thousands of events per second, fanning out updates to thousands of consumers, and achieving end-to-end delay under 10 seconds with 100% uptime. Kafka Streams was chosen for its millisecond delay guarantee and lightweight nature. The architecture involves an 'inflight spend' service that aggregates ad insertion events using Kafka Streams with a 10-second window store per adgroup. Key learnings include the performance impact of window store design (18x improvement by switching from hopping to tumbling windows) and message compression strategies (delta encoding and lookuptable encoding reducing message size by 4x).