Why you should move your ETL stack to Modal

What this post added

This post introduces and demonstrates Modal's capabilities for building and running ETL pipelines. It provides two concrete examples: 1) migrating 12 million rows from ClickHouse to Snowflake, highlighting cost savings compared to vendor solutions like Fivetran, and 2) enriching user data by querying the GitHub API, showcasing the ease of integrating custom code and external APIs. The post details the Python code used for extraction and loading, the use of Modal's `@app.function` decorator with secrets and timeouts, and the `for_each` method for parallel execution. It also shows how to schedule jobs using `modal.Cron`. The conclusion discusses when to use Modal for ETL versus traditional solutions like Fivetran or Airflow.