Sign Language Interface with MediaPipe

7 posts · Aug 2020 – Mar 2026

What this trail covers

The SignAll SDK provides developers with a sign language interface powered by MediaPipe, enabling the creation of applications that can interpret and process sign language gestures. This SDK leverages MediaPipe's capabilities for real-time perception and gesture recognition, opening up new possibilities for accessibility and human-computer interaction. Gemini's integration with MediaPipe further enhances this by allowing for natural language prompts to build interactive applications that leverag

All Google trails

Mar 2026Jump to play: Building with Gemini & MediaPipe- Google Developers BlogThis post introduces the integration of Gemini with MediaPipe for building interactive games and apps. It showcases how natural language prompts in Google AI Studio can generate functional web apps leveraging MediaPipe's on-device ML solutions for real-time input control. Examples include a motion-controlled Chrome Dino game using Pose Landmarker, a hair recoloring app using Image Segmenter, a hand gesture recognition app, and games like Bubble Gum Blow Challenge, Dalgona Candy, and Red Light, Green Light using Face Landmarker and Face Detector. It highlights the low latency benefits of on-device processing and the new long-range face detection capability in MediaPipe.
May 2024Project Gameface launches on Android- Google Developers BlogProject Gameface has been launched on Android, making its hands-free cursor control technology available for Android applications. This involves using the Android Accessibility Service to create a virtual cursor and leveraging MediaPipe's Face Landmarks Detection API, specifically its 52 face blendshape values, to map facial expressions and head movements to cursor control and actions. The implementation supports mobile actions like GLOBAL_ACTION_HOME, GLOBAL_ACTION_BACK, GLOBAL_ACTION_NOTIFICATIONS, and GLOBAL_ACTION_ACCESSIBILITY_ALL_APPS. A drag function has been implemented using the Android Accessibility Service to allow users to define start and end points for dragging actions. The camera feed is integrated to provide user feedback and maintain overlay visibility across the Android OS.
Jan 2024HealthPulse AI Leverages MediaPipe to Increase Health Equity- Google Developers BlogThis post details how HealthPulse AI, a digital health solution by Audere, leverages MediaPipe to create an AI-powered Rapid Diagnostic Test (RDT) reader. It highlights the use of MediaPipe's multi-platform support and efficient inference engine to run Computer Vision (CV) and Machine Learning (ML) models locally on low-end smartphones in decentralized care settings. The post describes the HealthPulse AI pipeline, which includes image quality assurance (IQA) to detect issues like blurriness and underexposure, RDT type detection, and classification of test results (positive, negative, invalid). It emphasizes how MediaPipe's graph definitions and calculators facilitate the development and iteration of this pipeline, enabling real-time feedback to users to retake photos when necessary. The contribution is the practical application of MediaPipe for medical diagnostics in resource-constrained environments, addressing critical global health challenges.
Oct 20237 dos and don'ts of using ML on the web with MediaPipe- Google Developers BlogThis post provides seven dos and don'ts for web developers using MediaPipe Solutions for machine learning. Key recommendations include not bundling ML models within the app but storing them in cloud storage and caching them locally, initializing tasks early to hide processing from the user, cleaning up WebAssembly resources using the .close() method, utilizing MediaPipe Studio for testing and customization, testing on different devices, and avoiding defaulting to the largest model, prioritizing speed and size based on use case.
Jun 2023Project GameFace makes gaming accessible to everyone- Google Developers BlogIntroduced Project Gameface, an open-source, hands-free gaming mouse utilizing MediaPipe's Face Landmarks Detection API with blendshape output. Detailed the use of forehead landmarks for cursor movement, separate cursor speed adjustment for different directions, and adjustable cursor smoothing to combat jitter. Explained the customization of facial expressions for mouse actions (click, scroll, reset cursor) and keyboard presses. Described the gesture size feature for triggering actions based on expression intensity. Highlighted the use of PyAutoGUI for mouse control, PyDirectInput for keyboard control, PyInstaller for packaging, and CustomTkinter for the UI.
Advertisement
Sep 2021Drone control via gestures using MediaPipe Hands- Google Developers BlogThis post details the integration of MediaPipe Hands, a machine learning solution for hand and finger tracking, with drone control systems. It demonstrates how hand gestures, detected by MediaPipe, can be translated into commands to control a drone's movement (e.g., forward, backward, up, down, rotation). This showcases a novel application of real-time gesture recognition for human-computer interaction in a physical control context.
May 2021Prosthesis control via Mirru App using MediaPipe hand tracking- Google Developers BlogThis post introduces the Mirru App, which utilizes MediaPipe's hand tracking capabilities to provide real-time prosthesis control. It demonstrates how MediaPipe can be used to interpret hand gestures for controlling external devices, highlighting the potential for AI-powered assistive technologies.
Apr 2021SignAll SDK: Sign language interface using MediaPipe is now available for developers- Google Developers BlogIntroduces the SignAll SDK, which integrates MediaPipe for sign language gesture interpretation and processing, enabling new accessibility and human-computer interaction applications. Mentions Gemini's integration with MediaPipe for building interactive applications via natural language prompts.
Aug 2020Instant Motion Tracking with MediaPipe- Google Developers BlogThis post introduces MediaPipe's new Instant Motion Tracking solution, which enables real-time motion tracking with low latency. It highlights the use of MediaPipe's Pose, Face, and Hands solutions to achieve this, demonstrating how these components can be combined to create a comprehensive motion tracking system. The post also provides a link to a demo and code repository for developers to explore and implement this functionality.