Processing ETL tasks with Ratchet

This post introduces the use of the Go library Ratchet for building ETL pipelines within Grab's Lending team. It details the concepts of data pipelines and ETL, explaining the extraction, transformation, and loading steps. The post highlights how Ratchet's Data Processors and Job Processors are used to manage these stages, with a specific example of whitelisting merchants using a CSV upload process. The transformation step includes data validation and API calls to external microservices, and the load step is omitted for this particular use case.