Confluent

Kafka Connect Sink for PostgreSQL from JustOne Database | Confluent

Jun 1, 2016Duncan Pauly
Kafka Connect Sink for PostgreSQL from JustOne Database | Confluent
SummaryAI, from the source post

This post introduces a new open-source Kafka sink connector for PostgreSQL-compatible databases. It details the use case of creating a relational data lake by streaming enterprise messages into a PostgreSQL database for agile analytics. Performance benchmarks show high ingestion rates (over 200,000 messages/second) into unindexed tables. The post explains the message-to-row conversion mechanism using JSON parsing paths and properties like `db.columns` and `db.json.parse`. It also covers database connection properties, target table definition, and delivery semantics (Fastest, Guaranteed, Synchronized), with Synchronized being the default for exactly-once delivery. Buffer size configuration and Java heap options are also discussed.

More from the Confluent Connectors Portfolio trail