Cloud Function Platform

11/20/2024

This post details how OpenArt scaled their Gen AI art platform on hundreds of GPUs using Modal. They were able to deploy complex, custom ComfyUI workflows programmatically by defining container images with custom nodes, wrapping functions with Modal decorators for GPU requirements and file mounts, and using `modal deploy`. Modal's autoscaling capabilities were crucial for handling their growing user base, allowing them to scale up to hundreds of GPU containers without manual intervention. This demonstrates the platform's ability to support high-code, custom ML pipelines and scale them effectively.

10/25/2024

This post elaborates on the challenges of volatile GPU demand for both inference and training workloads, highlighting the inefficiencies of fixed capacity provisioning. It introduces strategies for improving GPU utilization through demand pooling, supply pooling (across regions, GPU types, and cloud vendors), and fast multi-tenancy scaling (serverless infrastructure). It also touches upon demand smoothing techniques. Modal's specific contributions include investing in resource pool scaling, "bin packing" jobs via mixed-integer programming, building a specialized file system for fast initialization, and snapshotting CPU/GPU memory. The post positions Modal as a solution for flexible GPU consumption, enabling bursty jobs and instantaneous scaling of GPU-based cloud functions.

9/18/2024

This post details how Contextual AI integrated Modal's cloud function platform with GPUs to automate their CI pipelines. It describes the technical setup involving GitHub Actions triggering Modal Functions with custom `pytest` execution, leveraging Modal's ability to provision GPUs on-demand and cache custom environments. The implementation highlights the benefits of reduced latency, parallel execution, and cost savings compared to traditional CI solutions.

9/10/2024

This post details the integration of dlt and dbt into the Modal cloud function platform for building a cost-effective analytics stack. It provides specific examples of running dlt pipelines for data loading from Postgres to Snowflake, including advanced configurations like Modal Proxy and scaling with starmap. It also demonstrates dbt transformations for data analysis, highlighting the cost-effectiveness and flexibility of using Modal for these tasks.

8/16/2024

Introduced an interactive code playground that allows users to write and run Modal code directly in their browser. This is implemented using Modal Sandboxes for secure, isolated code execution. The backend spawns a `modal.Sandbox` with the user's code mounted, executes it using `modal run`, and streams logs back. The frontend uses CodeMirror and Xterm.js. This enhances the learning journey and provides a quick way to experiment with Modal features.

3/14/2024

This post details the technical implementation of Modal's `modal-http` service, which enables serverless functions to handle HTTP and WebSocket requests. It explains how HTTP requests are translated into Modal's internal function-calling mechanism, addressing the challenges of supporting compute-heavy and long-running workloads within a serverless context. The post discusses the differences between traditional serverless platforms and Modal's approach, highlighting the use of Rust for performance and the ability to handle large inputs/outputs and complex heterogeneous workloads.

2/21/2024

This post details how Suno used Modal's cloud function platform to scale inference and batch pre-processing to thousands of GPUs, significantly reducing their launch timeline. It highlights Modal's ability to dynamically manage compute for batch workflows without configuration files, and its support for exposing functions to HTTP traffic and chaining inference functions for end-to-end sequences. The post emphasizes Modal's auto-scaling capabilities for thousands of GPUs to handle variable demand, preventing the need for Suno to over-provision hardware or compromise user experience. Suno's experience demonstrates the platform's effectiveness in accelerating product launches by abstracting away infrastructure management.

2/6/2024

Introduces support for NVIDIA H100 GPUs on Modal, enabling users to leverage these high-performance accelerators for machine learning tasks. The post highlights the speedups H100s offer for training and inference compared to A100s, details their technical specifications (80GB DRAM, NVLink), pricing ($7.65/hour/GPU), and how to enable them via the `@app.function(gpu='H100')` decorator. It emphasizes Modal's autoscaling capabilities for cost-effective utilization of these powerful GPUs.

10/10/2023

This post announces the general availability of Modal, a cloud function platform. It details the platform's capabilities, including running Python code in the cloud, deploying GPU-accelerated inference endpoints, and executing large-scale batch jobs. It highlights the developer experience, emphasizing fast feedback loops and the elimination of YAML configuration. The post also mentions the underlying technical challenges, such as building a custom file system to achieve rapid code execution. Customer testimonials from Ramp, Substack, Suno, and Sphinx are included to showcase production use cases.