Modal

Run GPU jobs from Airflow with Modal

Jun 20, 2024
Run GPU jobs from Airflow with Modal
SummaryAI, from the source post

This post introduces two methods for integrating Modal with Airflow: calling deployed Modal functions using `Function.from_name` and `lookup`, and executing arbitrary Python code within Modal Sandboxes via a custom `ModalOperator`. The `ModalOperator` serializes Python functions and executes them in a sandbox, allowing for dynamic configuration of resources like GPUs. It also demonstrates passing data between Airflow tasks and Modal functions using Modal Dicts.

More from the Sandboxed Code Execution trail