The Hugo evolution: Engineering Grab's unified, one-click data ingestion platform with Apache Flink

The Hugo platform has been engineered with a custom automation layer and Apache Flink to unify data ingestion workflows. This evolution replaces siloed onboarding processes with a seamless, one-click experience for both MySQL CDC and Kafka ingestion. For MySQL CDC, Flink directly reads the database binlog, eliminating intermediary components like Kafka Connect and Sprinkler, reducing operational overhead from 4 to 2 core components, automating schema detection, and enabling instant onboarding. For Kafka ingestion, Flink dynamically retrieves Protobuf schemas from Confluent Schema Registry, removing hardcoding and manual stream registration, and enabling click-to-query functionality in the Hugo UI. This has reduced onboarding time for Kafka pipelines to approximately 6 minutes and for MySQL CDC pipelines to approximately 3 minutes, leading to a significant increase in pipeline adoption.