Virtual Reality (VR) and Augmented Reality (AR) Development

6/23/2026

This post details the engineering challenges and solutions for developing ultra-narrow steel-can batteries for Meta's AI glasses. It describes the shift from traditional wound electrode architectures to die-cut stacked layers to reduce impedance and enable higher peak power. It also highlights the importance of tight manufacturing tolerances for maximizing usable volume and energy density. The post further discusses the evolution of battery capacity and runtime across generations of smart glasses, including the system-level optimizations required to double runtime without changing battery chemistry. It also touches upon the complexities of dual-battery systems in Oakley Meta Vanguards and the power demands of the Meta Ray-Ban Display glasses.

9/29/2025

This post introduces AssetGen, a new foundation model for 3D assets, and discusses its development and training. It highlights the role of LLMs in VR and the ambition of generating entire 3D worlds from text prompts, building upon Meta's ongoing work in VR/AR development.

3/4/2025

This post details the application of multimodal AI to Meta's Ray-Ban Meta glasses, showcasing how AI models process speech, text, and images to provide real-time information and assistance to wearers. It highlights the development of foundational models for wearables, referencing research like AnyMAL, and discusses the unique engineering challenges of integrating AI into compact, consumer-facing devices.

7/18/2024

This post introduces Caddy, a new mixed reality CAD application developed by a team of mechanical engineers. It highlights the integration of Caddy with Llama 3, demonstrating the application of LLMs in a mixed reality design context. The post also mentions the team's journey of teaching themselves to code and their goal of disrupting the CAD software space.

9/12/2023

This post details the proactive security research conducted by Meta's Native Assurance team on the Meta Quest 2's Android-based OS. It describes the discovery and exploitation of a memory corruption vulnerability in the VR Runtime service, which allowed for arbitrary native code execution from an unprivileged application. The post outlines the VROS architecture, the VR Runtime service, its attack surface, and the technical details of exploiting a specific vulnerability to hijack control flow, ultimately contributing to improved security posture for Meta Quest products.

2/24/2023

This post discusses the ongoing development of AR hardware, emphasizing the challenge of creating devices that are smaller, lighter, and as powerful as existing VR headsets. It highlights the 'zero-to-one' innovation required for AR hardware components and the importance of diversity and embracing failure in this development process.

2/16/2023

This post details the engineering challenges and solutions involved in developing the Ray-Ban Stories smart glasses, focusing on integrating cameras, microphones, audio, and touch controls into a fashionable, lightweight form factor with long battery life. It highlights this development as a key step towards Meta's broader AR glasses roadmap.

2/13/2023

This post details the development of a cross-platform runtime for Meta's AR platform, focusing on performance optimization and device reach. Key technical contributions include splitting the monolithic runtime into smaller, configurable plugins to exclude unnecessary capabilities, enabling the AR engine to run on a wide variety of devices from mobile to VR.

3/18/2019

This post details the advancements in creating lifelike virtual avatars for VR, leveraging groundbreaking 3D capture technology and AI systems. It highlights the completion of dedicated capture facilities for face and body, enabling reconstruction of body structure and measurement of body motion at an unprecedented level of detail. This data is used to build an automated pipeline for photorealistic avatars that can be animated in real time using machine learning breakthroughs.

12/19/2018

Introduced DeepFocus, an AI-powered system using a novel end-to-end convolutional neural network with volume-preserving interleaving layers to achieve real-time, realistic retinal blur in VR. This addresses the computational demands of traditional methods and enables compatibility with various accommodation-supporting VR headsets.

10/12/2018

This post summarizes technical talks from Oculus Connect 5, detailing advancements in VR development. It covers bootstrapping social VR with Unity (invites, matchmaking, VoIP, P2P networking), optimizing rendering in VR with Unity's Scriptable Render Pipeline (SRP) for high frame rates and consistent resolution, using RenderDoc for debugging and performance optimization on Oculus Rift and Quest (limiting textures and geometry), applying deep learning for interactive actors that react to their environments, the Oculus Lipsync SDK 1.30 for audio-to-facial animation (cross-platform, online/offline generation, laughter detection, DSP acceleration), sound design for Oculus Quest (handling multiple sounds, spatial audio mapping), and advancements in the Oculus Rift PC SDK (adaptive compositor kickoff, high precision dithering, Asynchronous Spacewarp).

5/2/2018

This post announces the rebranding of React VR to React 360, reflecting a more accurate representation of its current usage and future roadmap. Key improvements include easier integration of 2D UIs in 3D space, enhanced support for 180° mono and stereo video with built-in transitions and faster loading, and improved playback performance, especially on lower-end mobile devices, due to runtime architecture changes.

9/21/2017

This post details the engineering challenges and solutions involved in bringing SLAM (Simultaneous Localization and Mapping) to Facebook's mobile AR platform. Key contributions include: 1. Developing a SLAM library optimized for diverse mobile devices, integrating features from ORB-SLAM, SVO, and LSD SLAM, and performing fine-grained instruction-level optimizations to achieve 60 Hz performance. 2. Addressing device-specific calibration needs for thousands of Android models, including focal length, principal point, distortion parameters, rolling shutter, autofocus, autoexposure, IMU calibration, and clock synchronization. 3. Reducing the SLAM library's binary size from ~40MB to under 1MB by refactoring and using common Facebook libraries. 4. Creating the WorldTracker API, an umbrella interface that combines SLAM with other tracking algorithms (like gyroscope-enhanced image-based tracking) to ensure robust object placement even when SLAM confidence is low. 5. Showcasing the application of this technology through an AR art project with artist Heather Day, demonstrating real-time scene geometry analysis for virtual object placement.

7/26/2017

This post details the application of AI-driven style transfer techniques to generate special effects for VR films, specifically for the "Alteration" VR experience. It addresses the technical challenges of applying style transfer to high-definition, stereoscopic, and 360-degree VR video, including memory and time constraints, and the subjective challenge of optimizing AI models against aesthetic opinions. The post describes retraining neural networks with larger image resolutions (768x768) for style transfer, the computational cost (120 GPU-hours for training, 550 CPU-hours for encoding), and the method for ensuring stereoscopic consistency by mapping style transfer from the left eye to the right eye using disparity maps. This demonstrates AI's maturity as a creative tool for filmmakers in VR.

4/20/2017

This post details the technical advancements in Meta's Surround 360 camera system, moving beyond stereo 360 to capture and process more complex immersive video. It describes the hardware design with 17 cameras, the stitching and rendering pipeline, and the challenges of handling higher resolutions and dynamic range. The system's ability to capture and process data for more advanced VR experiences is a key contribution.

4/19/2017

Introduced the 360 Capture SDK, enabling instant capture of PC VR experiences as 360 photos and videos. This SDK leverages cube mapping instead of traditional stitching, resulting in improved accessibility (works on baseline VR hardware), higher quality (captures at resolutions up to 4K), and faster capture speeds (under one second for VR-quality 360 video at 30 fps while maintaining 90 fps for the VR experience). The SDK is compatible with Unity, Unreal, and native game engines, and supports NVIDIA and AMD GPUs.

4/19/2017

This post announces the open-sourcing of React VR, a new library that allows developers to build virtual reality experiences on the web using JavaScript and standard web tools, expanding on the declarative programming style of React and React Native.

4/18/2017

Introduced React VR, a new library that allows developers to build virtual reality (VR) and 360-degree content using JavaScript and standard web tools. React VR leverages WebGL and WebVR APIs for immersive headset integration and is accessible on mobile phones and PCs. It allows composition of 3D scenes with 360 panoramas, 2D UI, text, images, audio, video, and 3D models. The library is built as a true React Native platform, reusing code and systems while extending them for browser and 3D space compatibility. It employs a web worker architecture, similar to React Native, to run React and rendering in separate contexts, with an asynchronous bridge for communication, ensuring smooth performance at 60-90 frames per second.

2/22/2017

This post details end-to-end optimizations for dynamic streaming of 360 video, building upon previous work. Key contributions include the introduction of variable offset cubemaps for encoding, which allows for more efficient pixel allocation towards the viewport, reducing overall resolution without sacrificing perceived quality. It also describes improvements in stream selection logic by considering six factors (resolution, video quality, current view orientation, stream viewport direction, length of the offset, and field of view) to decrease interruptions by over 80%. Further optimizations include more frequent bandwidth estimation, smarter handling of video chunks (playing fetching chunks only when no fallback is available, and canceling replacement chunks if a fallback is used), and prefetching initialization and index chunks for DASH streams. Player performance was boosted by filtering stream candidates early, pre-computing intensive computations, and using lookup tables.

10/6/2016

This post details Meta's exploration into building social experiences in VR, focusing on enhancing social presence through more expressive and representative avatars. It describes algorithmic approximations for arm and body position, realistic eye movement with blinks, voice-triggered mouth movement, and the display of emotions. The post also discusses the integration of interactive objects and the ability to share 2D/360 photos, videos, and Facebook posts within VR. Furthermore, it outlines methods for sharing VR experiences in real-time with non-VR users via Messenger video calls and capturing VR experiences for external sharing as selfies, videos, or 360 videos.

2/21/2016

Introduced dynamic streaming technology for 360 video to Gear VR, enabling more efficient delivery of VR video content. This technology aims to make videos look clearer and play faster for users.

6/19/2015

Introduced DynamicFusion, the first real-time dense dynamic scene reconstruction system, which allows for the reconstruction and tracking of non-rigid scenes. This technology removes the need for static subjects during scanning, opening up new applications for real-time 3D scanning and enhancing the integration of the real world into VR/AR.