Toward More Controllable AI Video Editing: An Early Research Exploration at Netflix

This post introduces two research explorations, Vera and VOID, aimed at improving controllability and physical plausibility in AI video editing. Vera is a layered video diffusion model that generates edits as separate layers, preserving original footage outside the edited regions. It uses a Mixture-of-Transformers architecture with separate DiTs for edit layer, alpha matte, and composite layer generation, trained on a custom layered video dataset. VOID is a video inpainting model for object and interaction deletion that reconstructs the scene plausibly. The post details the challenges of unintended edits and unnatural physics in current generative video editing models and presents Vera and VOID as solutions, releasing their research papers and code.