MediaPipe: Enhancing Virtual Humans to be more realistic- Google Developers Blog

What this post added

This post details the integration of MediaPipe Face Landmarker solution by KDDI to achieve real-time facial landmark detection and extract 52 blendshape scores for rendering realistic virtual human avatars. It provides a Python code example for using the MediaPipe Face Landmarker with blendshapes in LIVE_STREAM mode, storing blendshape data in Firebase Realtime Database, and transmitting it to Google Cloud's Immersive Stream for XR for real-time animation in Unreal Engine. It also outlines the Unreal Engine side implementation using the Firebase C++ SDK and the GameInstance Subsystem for receiving blendshape data and driving facial animations.