TechPrufer
Get started
Google
Google logoGoogle · Feature trail

Conversational Computing and Ubiquitous Interfaces

14 posts · Jul 2015 – May 2026

What this trail covers

Google's Home APIs are evolving to integrate Gemini's AI capabilities, enabling developers to build more intelligent and responsive smart home experiences. This includes Gemini-powered camera features for natural language querying of video clips, AI-driven automation suggestions, conversational automation drafting, and new triggers based on dates and weather. These advancements extend across Google's surfaces, including the Gemini app, smart speakers, smart displays, and Google TV, with a Home S. The Gemini API is also being leveraged to enable IoT devices to understand and react to custom speech commands, bridging the gap between the digital and physical worlds. This involves audio capture, encoding, API requests with function calling, AI processing, and response delivery to the device. Potential applications include smart home automation, robotics, and industrial IoT.

All Google trails
  1. Empowering Service Providers and Hardware Partners with Gemini for Home- Google Developers BlogThis post announces Gemini for Home, a full-stack AI offering that combines Google Home APIs with Gemini features to enable service providers and hardware manufacturers to build monetizable, proactive services. It details three core capabilities transforming the Google Home experience: camera intelligence for event description, 'Ask Home' for complex household questions, and 'Home Brief' for daily summaries. For service providers, it highlights opportunities to offer daily household awareness, advanced deterrence, and proactive protection. It also announces the expansion of the Google Home Gemini built in Program for hardware partners, offering validated reference designs for smart cameras and smart speakers. AT&T is cited as an early adopter integrating Gemini features into their security service.
  2. Gemini for Home: Expanding the Platform for a New Era of Smart Home AI- Google Developers BlogIntroduces Gemini for Home, replacing Google Assistant on speakers and displays, enhancing cameras, and overhauling the Google Home app. It expands the platform for Google Home by enabling new Gemini-powered features for existing devices through the Works with Google Home program and launching a new program for AI cameras. This program includes a hardware reference design, a Google Home camera embedded SDK with on-device ML, and a split compute model for camera intelligence.
  3. Bringing Gemini intelligence to Google Home APIs- Google Developers BlogThis post announces the integration of Gemini's AI capabilities into Google Home APIs. Key contributions include: 1. Expanding access to over 750 million devices. 2. Introducing Gemini-powered camera features for developers, including AI descriptions and history search. 3. Enhancing the Automations API with 'Suggested Automations', 'Help me create' conversational drafting, and new date/weather-based automation starters. 4. Demonstrating cross-surface integration of smart home devices with Gemini experiences. 5. Announcing an early access program for Gemini-powered Home APIs and developer challenges.
  4. Achieve real-time interaction: Build with the Live API- Google Developers BlogIntroduces the Gemini Live API, enabling real-time interaction with streaming audio, video, and text. Key new features include enhanced session management (longer sessions via context compression, session resumption, graceful disconnect notification), configurable turn control (voice activity detection, interruption handling), flexible session settings, expanded voice and language options for audio output, text streaming, and token usage reporting. Demonstrates real-world applications like Daily.co's Word Wrangler, LiveKit's AI copilot, and Bubba.ai's hands-free application for truck drivers.
  5. The Gemini API and the Internet of Things- Google Developers BlogThis post details how to leverage the Gemini REST API with IoT devices, specifically using an ESP32 microcontroller, to enable voice command control of devices. It outlines the process of audio capture, encoding (WAV to base64), API request with function calling (defining custom functions like `changeColor` and `toggleLights`), AI processing by Gemini, and receiving responses for device action. The example demonstrates turning an LED on/off and changing its color via voice commands, highlighting benefits like enhanced user experience, simplified command handling, and dynamic function execution based on user intent. It also mentions potential applications in smart home automation, robotics, and industrial IoT.
  7. Gemini 2.0: Level Up Your Apps with Real-Time Multimodal Interactions- Google Developers BlogIntroduced the Multimodal Live API for Gemini 2.0, enabling real-time, bidirectional streaming of text, audio, and video data via WebSockets. Key features include sub-second latency, natural voice conversations with interruption support, video understanding, tool integration within a single request, and steerable voices. This facilitates the development of interactive applications such as real-time virtual assistants and adaptive educational tools. Provided starter web application code and demo applications for exploration.
  8. Google Home is officially ready for your Matter devices and apps- Google Developers BlogThis post announces the general availability of Google Home for Matter devices and apps, marking a significant milestone in smart home interoperability. It highlights the integration of Matter devices into the Google Home ecosystem, enabling users to control and automate a wider range of smart home products. The post also touches upon the ongoing development of the Home APIs and runtime, emphasizing their role in facilitating new home experiences and AI-powered automation.
  9. Personalize user journeys by Pushing Dynamic Shortcuts to Assistant - Google Developers BlogThis post introduces the ability to push dynamic shortcuts to Google Assistant, enabling personalized user journeys. It details how developers can leverage this capability to proactively offer relevant actions to users based on their context and predicted needs, thereby enhancing user engagement and streamlining interactions with Assistant-powered applications.
  10. Improve your development workflow with Interactive Canvas DevTools - Google Developers BlogThis post introduces Interactive Canvas DevTools, a new set of debugging and testing tools for developers building conversational experiences with Google Assistant. These tools allow developers to inspect and debug their interactive canvas applications, including the ability to test voice interactions and view the state of the application in real-time. This enhances the developer workflow for creating rich, conversational interfaces.
  11. Announcing New Smart Home SHED Types and Traits- Google Developers BlogThis post announces the introduction of new Smart Home SHED (Smart Home Enablement Device) types and traits for Google Home APIs. These additions expand the range of devices and functionalities that developers can integrate into the Google Home ecosystem, enabling more sophisticated smart home experiences. The new types and traits allow for finer-grained control and richer interactions with a wider variety of smart home devices.
  12. The Google Assistant SDK - new languages, API updates, and Device Actions- Google Developers BlogThis post announces the expansion of the Google Assistant SDK to include new languages (Spanish, French, German, Italian, Japanese, Korean, Portuguese, and Russian) and API updates. It also introduces Device Actions, allowing developers to integrate their devices with Google Assistant for voice control. This includes support for device commands, queries, and the ability to create custom device actions.
  13. Introducing the Google Assistant SDK- Google Developers BlogThis post introduces the Google Assistant SDK, enabling developers to integrate Google Assistant's conversational capabilities into their own devices and products. It allows for programmatic control of Assistant features, including voice input, audio output, and device actions, facilitating the creation of custom voice-controlled experiences.
  14. Connect With the World Around You Through Nearby APIs- Google Developers BlogIntroduces the Nearby API, a new developer API that enables device-to-device communication and interaction, allowing apps to discover and connect with other nearby devices. This facilitates real-time, local interactions for a variety of use cases, including multiplayer gaming, file sharing, and collaborative experiences, without relying on an internet connection.