Run GPU jobs from Airflow with Modal
Jun 20, 2024

Summary
This post introduces two methods for integrating Modal with Airflow: calling deployed Modal functions using `Function.from_name` and `lookup`, and executing arbitrary Python code within Modal Sandboxes via a custom `ModalOperator`. The `ModalOperator` serializes Python functions and executes them in a sandbox, allowing for dynamic configuration of resources like GPUs. It also demonstrates passing data between Airflow tasks and Modal functions using Modal Dicts.
More from the Sandboxed Code Execution trail
- VM Sandboxes: Full computers for agents | Modal Blog
- Sidecars: A low-latency trust boundary for Sandboxes | Modal Blog
- Runtime Roundup: VM Sandboxes, Multi-node clusters, and more | Modal Blog
- Product updates: Sandbox Sidecars, new models, a refreshed dashboard, and more | Modal Blog
- Devin Outposts on Modal | Modal Blog