Clips Background Effects

1 post · Jun 2022

What this trail covers

This thread tracks the evolution of Slack's Clips feature, focusing on enhancing video communication with background effects. Initial posts detailed the release of Clips and the addition of thumbnail selection and background blur. This post details the technical implementation of background blur and background image replacement for browsers and the desktop client, leveraging WebGL and WebAssembly for performance. It covers the use of the MediaPipe selfie segmentation model, bilateral smoothing, and specific techniques for achieving crisp edges in background replacement.

All Slack trails

Jun 2022Building Background Effects for ClipsThis post details the technical implementation of background effects (blur and replacement) for Slack Clips. It covers the use of the Media Streams API and Insertable Streams API for real-time video processing, offloading to a worker thread. The core of the implementation involves using Google's MediaPipe selfie segmentation model (TensorFlow Lite via WebAssembly) to generate an alpha mask, followed by bilateral smoothing to refine edges. For background blur, a weighted Gaussian blur is applied using the mask as a weight. For background replacement, a multi-step process is used: blurring the mask with an inverted parabola weighting, compressing the blurred mask to create sharp edges, blurring the background image based on the mask, and finally alpha blending the foreground and background. The post also discusses exploring edge detection methods and acknowledges the use of WebGL and WebAssembly for performance.
Advertisement