Improving Hugo stability and addressing oncall challenges through automation

This post details the implementation of an automation solution for Grab's Hugo data ingestion platform to improve stability and reduce on-call challenges. The solution introduces six core modules: Signal (collecting failure, SLA, and data quality signals), Diagnosis (determining root cause, assignee, and resolution method), RCA Table (storing diagnosis information), Auto-resolution (executing automated fixes), Data Health API (providing health status to external systems like Kinabalu), and the Hugo pipeline health dashboard (Data Health Workbench) for visualization and manual intervention. The diagnosis module leverages parallelism to handle surges in errors and avoids parsing extensive logs by relying on signals from computation engines. The auto-resolution module employs a flexible framework with custom handlers, including retry mechanisms with backoff. The impact includes enhanced data visibility, timely issue resolution, reduced on-call workload, and improved data contract management.