Cloud Function Platform

9 posts · Feb 2024 – Nov 2024

What this trail covers

Modal's cloud function platform supports automated CI workflows on GPUs, enabling parallelized testing and faster iteration for AI development. This integration allows GitHub Actions to trigger Modal Functions with custom GPU requirements, significantly reducing test execution time and eliminating the need for self-hosted runners. The platform's usage-based billing and cached image builds contribute to cost-effectiveness.

All Modal trails

Nov 2024How OpenArt scaled their Gen AI art platform on hundreds of GPUsThis post details how OpenArt scaled their Gen AI art platform on hundreds of GPUs using Modal. They were able to deploy complex, custom ComfyUI workflows programmatically by defining container images with custom nodes, wrapping functions with Modal decorators for GPU requirements and file mounts, and using `modal deploy`. Modal's autoscaling capabilities were crucial for handling their growing user base, allowing them to scale up to hundreds of GPU containers without manual intervention. This demonstrates the platform's ability to support high-code, custom ML pipelines and scale them effectively.
Oct 2024The future of AI needs more flexible GPU capacityThis post elaborates on the challenges of volatile GPU demand for both inference and training workloads, highlighting the inefficiencies of fixed capacity provisioning. It introduces strategies for improving GPU utilization through demand pooling, supply pooling (across regions, GPU types, and cloud vendors), and fast multi-tenancy scaling (serverless infrastructure). It also touches upon demand smoothing techniques. Modal's specific contributions include investing in resource pool scaling, "bin packing" jobs via mixed-integer programming, building a specialized file system for fast initialization, and snapshotting CPU/GPU memory. The post positions Modal as a solution for flexible GPU consumption, enabling bursty jobs and instantaneous scaling of GPU-based cloud functions.
Sep 2024How Contextual AI automated CI with Modal GPUsThis post details how Contextual AI integrated Modal's cloud function platform with GitHub Actions to automate their CI pipelines on GPUs. It highlights the use of Modal Functions with custom GPU requirements and pytest for running test suites, enabling parallel execution of CI jobs for multiple pull requests. Key benefits include eliminating self-hosted runners, fast image caching for quick test startup (within 30 seconds), and usage-based billing. The post also provides a simplified code example of a Modal Function for pytest execution.
Sep 2024Building a cost-effective analytics stack with Modal, dlt, and dbtThis post details the integration of dlt and dbt into the Modal cloud function platform for building a cost-effective analytics stack. It provides specific examples of running dlt pipelines for data loading from Postgres to Snowflake, including advanced configurations like Modal Proxy and scaling with starmap. It also demonstrates dbt transformations for data analysis, highlighting the cost-effectiveness and flexibility of using Modal for these tasks.
Aug 2024Inside the Modal code playgroundIntroduced an interactive code playground that allows users to write and run Modal code directly in their browser. This is implemented using Modal Sandboxes for secure, isolated code execution. The backend spawns a `modal.Sandbox` with the user's code mounted, executes it using `modal run`, and streams logs back. The frontend uses CodeMirror and Xterm.js. This enhances the learning journey and provides a quick way to experiment with Modal features.
Advertisement
Mar 2024Lambda on hard mode: Inside Modal's web infrastructureThis post details the technical implementation of Modal's `modal-http` service, which enables serverless functions to handle HTTP and WebSocket requests. It explains how HTTP requests are translated into Modal's internal function-calling mechanism, addressing the challenges of supporting compute-heavy and long-running workloads within a serverless context. The post discusses the differences between traditional serverless platforms and Modal's approach, highlighting the use of Rust for performance and the ability to handle large inputs/outputs and complex heterogeneous workloads.
Feb 2024How Suno shaved 4 months off their launch timeline with ModalThis post details how Suno used Modal's cloud function platform to scale inference and batch pre-processing to thousands of GPUs, significantly reducing their launch timeline. It highlights Modal's ability to dynamically manage compute for batch workflows without configuration files, and its support for exposing functions to HTTP traffic and chaining inference functions for end-to-end sequences. The post emphasizes Modal's auto-scaling capabilities for thousands of GPUs to handle variable demand, preventing the need for Suno to over-provision hardware or compromise user experience. Suno's experience demonstrates the platform's effectiveness in accelerating product launches by abstracting away infrastructure management.
Feb 2024Introducing: H100s on ModalIntroduces support for NVIDIA H100 GPUs on Modal, enabling users to leverage these high-performance accelerators for machine learning tasks. The post highlights the speedups H100s offer for training and inference compared to A100s, details their technical specifications (80GB DRAM, NVLink), pricing ($7.65/hour/GPU), and how to enable them via the `@app.function(gpu='H100')` decorator. It emphasizes Modal's autoscaling capabilities for cost-effective utilization of these powerful GPUs.