Running Kafka Streams Applications in AWS | Confluent

What this post added

This post details Zalando's experience running Kafka Streams applications in production on AWS. It covers challenges and solutions related to AWS EBS I/O throttling, fine-tuning Kafka Streams configurations (e.g., max.poll.records, processing timeouts), and managing JVM heap and off-heap memory usage with RocksDB. It also introduces custom monitoring tools like Remora for consumer lag and discusses the importance of monitoring EBS burst balance, JVM heap, CPU, system memory, and disk usage.