Scheduling Jupyter Notebooks at Meta

This post details the development of a scheduled notebooks infrastructure for Meta's Bento Jupyter notebooks platform, integrating it with the Dataswarm batch ETL pipeline framework. Key technical contributions include the BentoOperator, which executes notebooks in network-isolated containers, uses custom cells for structured data fetching (e.g., SQL cells) and writing (e.g., pandas.read_csv/to_csv), and leverages input/output data annotations for lineage tracking. The system enforces privacy through integration with Meta's data purpose framework and a "purpose policy zone." The workflow involves UI for schedule setup, diff creation for review, and a debugging interface. Future work aims to expand supported notebook types, introduce parameterized notebooks, and implement event-based scheduling.