ETL Platform

1/15/2026

This post highlights how Chai Discovery leverages Modal Volumes for efficient handling of large biological datasets (hundreds of gigabytes) in their ETL pipelines. Previously, these datasets required hours of downloading and indexing per machine. With Modal Volumes, the data is downloaded and indexed once, then instantly shared across all machines, enabling near-instant cold-start attachment and consistent performance. This significantly reduces data setup time for multiple sequence alignment (MSA) workloads and other bioinformatic computations, allowing researchers to start new runs immediately without repeated setup or storage overhead.

12/28/2024

Introduced support for eStargz compression format to speed up image builds from external registries. Added OpenID Connect (OIDC) integration for authenticating against external resources like S3 buckets and secret managers, offering an alternative to Modal Secrets. Enhanced memory snapshotting for faster cold starts, particularly for models like Stable Diffusion. Expanded capabilities for batch processing and job queues with a significantly higher input limit (up to 1 million inputs).

12/4/2024

This post details how a European soccer team used Modal for processing spatio-temporal match data on GPUs. It highlights the use of Modal's serverless batch processing capabilities, rapid container startup times (seconds vs. minutes), usage-based pricing leading to a 50% cost reduction, and automatic scaling for high parallelization. The team also built an in-memory vector database on Modal for semantic search with embeddings, demonstrating cost-effectiveness compared to managed solutions. The post also mentions the ease of integrating with cloud storage and switching between CPU and GPU types.

4/18/2024

This post introduces and demonstrates Modal's capabilities for building and running ETL pipelines. It provides two concrete examples: 1) migrating 12 million rows from ClickHouse to Snowflake, highlighting cost savings compared to vendor solutions like Fivetran, and 2) enriching user data by querying the GitHub API, showcasing the ease of integrating custom code and external APIs. The post details the Python code used for extraction and loading, the use of Modal's `@app.function` decorator with secrets and timeouts, and the `for_each` method for parallel execution. It also shows how to schedule jobs using `modal.Cron`. The conclusion discusses when to use Modal for ETL versus traditional solutions like Fivetran or Airflow.