ETL Pipeline Orchestration with Ratchet

Grab's Lending team utilizes the Go library Ratchet for building custom data pipelines and handling ETL tasks. This system orchestrates batch processing, restructuring and rescheduling of loans, and creating user profiles. The Ratchet library, written in Go, allows for concurrent processing of data through Data Processors organized into stages within a pipeline. The Azkaban service manages these pipelines, with Job Processors defining the Extract, Transform, and Load stages. A specific use case involves whitelisting merchants by processing CSV files uploaded via an internal portal, extracting data from S3, transforming it through validation and API calls, and skipping the load step as transformed data is not stored.