Apache Kafka Lag Monitoring and Metrics at AppsFlyer

This post details AppsFlyer's implementation of a Kafka lag monitoring system using Burrow. It explains the challenges of monitoring Kafka lag in a large-scale event-driven architecture, the basic concepts of Kafka offsets and lag, and the decision to use Burrow due to its modular design, flexibility in metric calculation, and ability to support both __consumer_offsets and ZooKeeper consumers. The post also touches upon the importance of visibility in distributed systems and the need for actionable metrics to quickly diagnose and resolve issues.