Bottled Water: Real-time integration of PostgreSQL and Kafka | Confluent

This post introduces Bottled Water, an open-source tool for PostgreSQL that leverages logical decoding to transform database changes into a stream of structured Kafka events. It addresses the challenges of real-time data integration by providing a consistent snapshot and a real-time stream of changes, enabling downstream systems to stay up-to-date with minimal latency. The tool encodes data in Avro format and sends it to Kafka, where each table becomes a topic and each row becomes a message. It utilizes Kafka's log compaction feature to manage data, retaining the latest message for a given key indefinitely and garbage-collecting older messages for deleted or updated rows. This allows Kafka to serve as both a snapshot and a real-time message store.