Streamline Data Ingestion with Google BigQuery Sink V2 Connector on Confluent

What this post added

Introduces the BigQuery Sink V2 Connector, a significant upgrade to the existing BigQuery Sink Connector. Key technical enhancements include the adoption of Google's Storage Write API for improved efficiency and cost-effectiveness, OAuth 2.0 support for enhanced security, parallel record streaming through an internal thread pool for greater scalability, and support for multiple data formats (Avro, JSON Schema, Protobuf, JSON). It also adds time-based table partitioning and invalid record routing to a DLQ. Migration recommendations from the legacy connector are provided.