Putting sign language AI into users’ hands

This post introduces the massively multilingual sign-language-to-text (SL2T) translation model. It details the challenges of sign language translation compared to spoken language transcription, emphasizing the need for true machine translation and complex visual perception. The SL2T model is trained on over 100,000 hours of data across more than 50 sign languages. A key technical detail is the use of pose landmark locations instead of raw camera feeds to protect user privacy, with an on-device MediaPipe Holistic model tracking signer points. The model bypasses intermediate 'glosses' and translates directly from landmarks to text, achieving a zero-shot BLEURT score of 70 on FLEURS-ASL. Practical considerations like streaming latency, hallucination prevention, and fairness for left-handed and one-handed signing are also addressed.