Scheduling Jupyter Notebooks at Meta

What this post added

Introduced a scheduled notebooks infrastructure by integrating the internal Bento Jupyter notebooks platform with the Dataswarm batch ETL pipeline framework. This system enables automated execution of notebooks for ETL, data transformations, and visualizations. Key technical contributions include the development of a custom BentoOperator that runs notebook executions in network-isolated containers, uses custom cells for structured data fetching (e.g., SQL cells) and writing (e.g., pandas.DataFrame.to_csv()), and integrates with Meta's data purpose framework for privacy compliance. The system also includes UI for schedule setup, a diff review process, and a debugging interface.