Building Background Effects for Clips

This post details the technical implementation of background effects (blur and replacement) for Slack Clips. It covers the use of the Media Streams API and Insertable Streams API for real-time video processing, offloading to a worker thread. The core of the implementation involves using Google's MediaPipe selfie segmentation model (TensorFlow Lite via WebAssembly) to generate an alpha mask, followed by bilateral smoothing to refine edges. For background blur, a weighted Gaussian blur is applied using the mask as a weight. For background replacement, a multi-step process is used: blurring the mask with an inverted parabola weighting, compressing the blurred mask to create sharp edges, blurring the background image based on the mask, and finally alpha blending the foreground and background. The post also discusses exploring edge detection methods and acknowledges the use of WebGL and WebAssembly for performance.