How Apache Kafka Enables Podium to "Ship It and See What Happens"

This post details how Podium migrated their conversation search functionality from a Postgres read replica to Elasticsearch using a Kafka-based event streaming pipeline. It covers the use of Debezium PostgreSQL CDC Connector to ingest data into Kafka, ksqlDB for transforming and calculating conversation steps, joining records from different topics (users, conversations, conversation_items), and using tombstones to remove closed conversations from Elasticsearch. It also discusses factors like restreaming for idempotency and versioning topics for data evolution.