TechPrufer
Get started
Modal
Modal logoModal · Feature trail

AI Research Infrastructure

7 posts · May 2024 – Sep 2026

What this trail covers

Modal's platform provides scalable, reproducible, and isolated infrastructure for executing AI research workloads, including GPU compute, consistent benchmarking environments, and sandboxed execution. This enables researchers to accelerate AI development, particularly in areas like automated kernel generation and self-improving AI systems. The platform now offers enhanced support for Reinforcement Learning (RL) post-training of LLMs, addressing infrastructure challenges in training, rollouts, and deployment. Substack has migrated both training and deployment of their ML models from AWS SageMaker to Modal, significantly improving developer experience, collaboration, and iteration speed. They implemented a full fine-tune → validation → deployment workflow leveraging Modal's native storage primitives and key-value store for tracking production model versions. This has enabled faster development and deployment of ML workflows, including a new recommendation model in days.

All Modal trails
  1. How Botika runs full-stack generative AI on Modal | Modal BlogThis post details how Botika leverages Modal for their full-stack generative AI operations, including a 100-terabyte data pipeline with thousands of concurrent containers across various GPU types, rapid iteration for foundation model training (including multi-node training and RL infrastructure), and scalable production inference for ~15 models. It highlights Modal's ability to handle complex AI workloads without requiring traditional cloud infrastructure management, emphasizing ease of use, speed, and scalability for both research and production.
  2. Anthropic integration with Modal brings scalable compute to Claude Science | Modal BlogThis post details the integration of Modal's AI research infrastructure with Anthropic's Claude Science, enabling life sciences researchers to run demanding computational workloads directly from a conversational interface. It highlights how Modal provides scalable compute for tasks like data processing pipelines, structure prediction models, and molecule design campaigns. Specific technical capabilities emphasized include fan-out for highly parallel workloads (e.g., virtual screening), GPU access per step for heterogeneous pipelines, shared storage via Modal Volumes for large datasets, and reproducible environments via Modal Images. The post also provides examples of how this integration facilitates complex biological analyses, such as engineering enzymes, designing genome-wide CRISPR knockout screens, and performing single-cell analysis at scale.
  3. Reinforcement learning is an infrastructure problem | Modal BlogThis post details Modal's infrastructure support for Reinforcement Learning (RL) post-training of LLMs. It highlights the challenges in training, rollouts, and isolated environment execution. Key contributions include enabling multi-node training with RDMA, improving weight synchronization with delta compression, and optimizing sandbox buffer sizing for GPU utilization. The post also emphasizes the integration with open-source RL frameworks and Modal's ability to manage compute capacity efficiently, allowing users to go from zero to a B200 cluster within minutes.
  4. Scaling reinforcement learning at Applied Compute | Modal BlogThis post details how Applied Compute uses Modal's sandboxes to create complex, production-mimicking environments for RL training, emphasizing fidelity and determinism. It highlights Modal's performance optimizations, specifically fast container image caching and sub-second cold starts, which keep GPUs utilized by minimizing CPU-bound idle time during rollouts. The post also explains the use of Modal Functions for massively parallel grading, leveraging automatic retries and per-invocation isolation for reliability under load. This contributes to the AI Research Infrastructure capability by demonstrating its application in a complex RL training loop.
  5. Building an RL theorem-proving workflow on Modal | Modal BlogThis post details the implementation of an RL theorem-proving workflow on Modal using Evolution Strategies (ES). It highlights the use of separate container images for GPU inference (vLLM) and CPU-based Lean verification, parallel GPU fan-out with `.map()` for ES iterations, and sandboxed Lean verification for isolation. It also describes stateless checkpointing using seed histories and volumes for base model storage, reducing download times. The implementation achieved a 60% reduction in project completion time compared to alternative platforms.
  7. Autoscaling Autoresearch: Give your agents elastic GPUs on Modal | Modal BlogThis post details the application of Modal's elastic GPU provisioning and autoscaling capabilities to AI research agents, specifically in the context of OpenAI's Parameter Golf challenge. It demonstrates how agents can dynamically acquire and release compute resources (from single GPUs to multi-GPU clusters) based on the needs of different research stages (validation, exploration, validation at scale, debugging, and optimization). The post highlights the speedup and efficiency gains achieved by using Modal's serverless runtime, which allows for rapid scaling from one GPU to dozens or hundreds as a parameter change, and automatic scaling to zero when work is completed. It also showcases the use of Modal Skills to guide agents in provisioning GPUs, running experiments, and managing storage.
  8. How Doppel eliminated ML infrastructure tax with Modal | Modal BlogThis post details how Doppel migrated their ML training and inference workflows to Modal. For training, they leveraged Modal's `map()` function to parallelize K-fold cross-validation, significantly reducing experimentation time compared to sequential runs. They also integrated coding agents with Modal's CLI to automate mechanical steps in the experimentation loop, such as launching experiments and collecting metrics. For inference, Doppel migrated from a GCP Cloud Run stack to Modal, reducing model build times by up to 10x through image layer caching and persistent volumes. Modal's serverless architecture automatically scales inference workloads to handle traffic spikes, and it eliminates the need for custom HTTP service layers around models, simplifying deployment and reducing integration code.
  9. Accelerating AI research that accelerates AI research | Modal BlogThis post details how Modal's platform is used by academic researchers to accelerate AI research that itself accelerates AI research. It highlights the use of Modal for benchmarking GPU kernels (KernelBench), writing better GPU kernels than human engineers (TTT-Discover), and pursuing the automation of the entire machine learning lifecycle (RL-4-MLE). Specific technical contributions include: consistent benchmarking across hardware configurations and software environments, scalable parallel execution of kernel compilation and benchmarking on hundreds of concurrent GPUs with minimal code modification, and providing vetted, dependable, and isolated GPUs for RL systems sensitive to nondeterminism and hungry for compute.
  10. Introducing Notebooks – launch ML experiments with zero cold bootsIntroduces Modal Notebooks, a collaborative environment for high-performance interactive computing. Key features include: fast cold-start times (under 5 seconds) on arbitrary container images and hardware up to 8 H100/B200 GPUs; auto-idling and resuming kernels to optimize cost; unified access to Volumes, Secrets, and deployed Functions; and real-time collaborative editing. Notebooks integrate with the Modal stack, allowing direct calls to Modal Functions and reuse of existing images and runtime primitives. They also offer modern dev experiences with Pyright language server and AI completions. The post highlights industry usage and upcoming features like memory snapshots and scheduled runs.
  11. Product updates: Multi-node training clusters, B200 and H200s, and Client 1.0 releaseIntroduced multi-node training clusters with the `@clustered` decorator, enabling linear scaling of training runs across dozens of GPUs on multiple hosts via high-speed RDMA interconnect. Added support for NVIDIA B200 and H200 GPUs for serverless LLM inference, offering significant speedups over H100s. Released version 1.0 of the Modal client, emphasizing API stability and predictability, with specific updates including `modal.Volume.read_only`, `--secret` option for `modal shell`, timezone support for `Cron` schedules, and a `--timestamps` flag for `modal app logs`.
  12. Why Substack moved their AI and ML pipelines to ModalThis post details Substack's migration of their AI and ML training and deployment pipelines from AWS SageMaker to Modal. Key technical contributions include: 1. Improved developer experience by allowing development and testing directly in the code editor and command line, eliminating the need for convoluted SageMaker product navigation and remote environment juggling. 2. Enhanced collaboration by enabling code sharing across projects, avoiding duplication previously seen with SageMaker's code packaging. 3. Reduced container startup times, moving from 5+ minutes on SageMaker to near-instantaneous startup on Modal, facilitating rapid iteration. 4. Implementation of a full fine-tune → validation → deployment workflow using Modal's native storage primitives (mounting S3 buckets, saving weights to Modal network volumes) and a Modal key-value store for tracking production model versions. This workflow significantly accelerated the development and deployment of ML models.