TechPrufer
Get started
Modal
Modal logoModal · Feature trail

ETL Platform

4 posts · Apr 2024 – Jan 2026

What this trail covers

Modal's platform supports eStargz compression for faster image builds from external registries like ECR and Docker Hub, and introduces OpenID Connect (OIDC) for secure authentication against external resources, reducing the need for long-lived credentials. The platform also enhances memory snapshotting for faster cold starts and expands capabilities for batch processing and job queues with a higher input limit. This post highlights the use of Modal for large-scale spatio-temporal data processing and demonstrates cost-effective ETL pipelines by comparing custom code on Modal against vendor solutions like Fivetran, showcasing significant cost savings for large data volumes and flexible custom code execution.

All Modal trails
  1. Seamless computational bio at Chai Discovery | Modal BlogThis post highlights how Chai Discovery leverages Modal Volumes for efficient handling of large biological datasets (hundreds of gigabytes) in their ETL pipelines. Previously, these datasets required hours of downloading and indexing per machine. With Modal Volumes, the data is downloaded and indexed once, then instantly shared across all machines, enabling near-instant cold-start attachment and consistent performance. This significantly reduces data setup time for multiple sequence alignment (MSA) workloads and other bioinformatic computations, allowing researchers to start new runs immediately without repeated setup or storage overhead.
  2. Modal + Datalab: Deploy high-throughput document intelligence in <5 minutesThis post details how to deploy Datalab's Marker and Surya document intelligence models on Modal. It showcases caching model weights in Modal Volumes for faster cold starts, specifying environment and hardware requirements in function decorators, and leveraging Modal's autoscaling for high throughput on batch jobs. It highlights a 10x throughput gain for Marker on Modal compared to local CPU processing, achieved through GPU utilization, Flash Attention optimizations, and environment tuning. The post also mentions Datalab's managed API platform, which uses Modal for inference.
  3. Product updates: memory snapshotting, OIDC, async job queues & moreIntroduced faster memory snapshotting for models like Stable Diffusion, cutting cold start times by half. Expanded async job queues to support up to 1 million inputs. Added OpenID Connect (OIDC) for short-lived token authentication against external resources, replacing Modal Secrets. Enabled eStargz compression for faster image builds from external registries. Updated CLI with `--detach` flag for `modal launch` and added streaming output, timeout, workdir, and secrets parameters to `Sandbox.exec`.
  4. How a top tier European soccer team sped up their data processing and reduced costs by 50%This post details how a European soccer team (AFC Richmond) used Modal for processing spatio-temporal match data. The solution involved ingesting tracking data, running inference with a custom transformer-based model on GPUs, and writing results to cloud storage. Modal's serverless, usage-based pricing and fast container startup times led to a 50% cost reduction and processing times reduced from hours to minutes. The team also built an in-memory vector database on Modal for semantic search, demonstrating cost-effectiveness compared to managed solutions.
  5. Building a cost-effective analytics stack with Modal, dlt, and dbtThis post details the integration of dlt and dbt within Modal to build a complete analytics stack. It demonstrates how to use dlt for data loading from a Postgres read replica to Snowflake, including automatic schema inference, Snowflake DDL generation, and incremental loading. It also shows how dbt can be used for data transformation, materializing models as views. The post highlights cost savings compared to vendor solutions and provides project structure and code examples for setting up CI/CD with GitHub Actions.
  7. Why you should move your ETL stack to ModalThis post demonstrates how to build cost-effective and flexible ETL pipelines using Modal. It provides two concrete examples: 1) Copying 12 million ClickHouse rows to Snowflake for $0.29, highlighting Modal's cost savings over vendor solutions like Fivetran for large data transfers. 2) Enriching user data by querying the GitHub API, showcasing the ease of integrating custom code and scheduling jobs with Modal's `@app.function` decorator and `modal.Cron` schedule. The post also discusses when traditional ETL solutions or orchestration platforms like Airflow might be more suitable for long-running, business-critical, multi-stage pipelines.