Powering Talking Characters with Generative AI- Google Developers Blog

This post introduces a customizable AI-powered character template that demonstrates the power of LLMs to create interactive experiences with depth. It showcases how combining Large Language Models with existing Google APIs and technologies can solve specific industry use cases. The 'Talking Character' template is a 3D avatar builder that allows developers to bring an animated character to life with Generative AI, enabling configuration of personality, backstory, and knowledge base. Interactions are powered by Google's speech-to-text service, the PaLM API, and text-to-speech. Animation is achieved using the MediaPipe framework and a new audio-to-blendshapes machine learning model trained with TensorFlow, which predicts blendshape weights from speech input in real-time to drive the animated avatar. The model is trained from 'talking head' videos using 3D face tracking to learn a mapping from speech to facial blendshapes. Blendshape weights are then used to morph facial expressions and lip motion of the 3D avatar using the three.js library. Character design focuses on emotional bonding through backstory and personality. The prompt structure includes personality, backstory, and knowledge base, which are combined into a large prompt sent to the PaLM API. Partnerships with ZEPETO are highlighted, showcasing metaverse applications, and other use cases in gaming, education, and commerce are discussed.