MediaPipe: Enhancing Virtual Humans to be more realistic- Google Developers Blog

This post details the integration of MediaPipe's Face Landmarker solution with KDDI's virtual human 'Metako'. It showcases how MediaPipe's Python package can detect facial features and extract 52 blendshapes in real-time. The blendshape values are stored in Firebase Realtime Database and then transmitted to Google Cloud's Immersive Stream for XR instances. The Firebase C++ SDK is used on the Unreal Engine side to receive blendshape values, driving face animation in real-time via the 'Modify Curve' node in the animation blueprint. The GameInstance Subsystem is suggested as a pattern for managing the BlendshapesReceiver instance.