Simplified Dataflow Connectors with Managed I/O- Google Developers Blog

Introduces Managed I/O for Google Cloud Dataflow, which automates the management and upgrading of Apache Beam I/O connectors. This feature addresses complexities such as manual Beam upgrades, varying connector APIs, and sub-optimal connector performance on the Dataflow runner. Managed I/O provides automatic SDK upgrades, a simplified and standardized API for sources and sinks (e.g., `Managed.read(SOURCE).withConfig(sourceConfig)`), and automatic optimization for the Dataflow runner. Benchmarks demonstrate linear scaling for batch pipelines with Managed Iceberg I/O and high throughput (250k msgs/sec) for streaming pipelines with Managed Kafka I/O, while efficiently utilizing VM resources.