TechPrufer
Get started
Meta
Meta logoMeta · Feature trail

Virtual Reality (VR) and Augmented Reality (AR) Development

23 posts · Jun 2015 – Jun 2026

What this trail covers

Meta's engagement with Virtual Reality (VR) and Augmented Reality (AR) development has evolved significantly, encompassing the creation of dedicated hardware like Oculus Quest, advancements in rendering pipelines, performance optimization tools, and the integration of deep learning for interactive actors and facial animation. This includes sophisticated audio design for immersive experiences, improvements to PC SDKs, and now, the development of next-generation AR hardware that is smaller, lighter, and more powerful. The introduction of AssetGen, a foundation model for 3D assets, further enhances content creation capabilities by enabling the generation of 3D assets and entire virtual worlds from text prompts, leveraging LLMs and AI. This represents a significant step towards democratizing 3D content creation for VR and AR experiences.

All Meta trails
  1. How Meta Engineered Ultra-Narrow Batteries for AI GlassesThis post details the engineering challenges and solutions for developing ultra-narrow steel-can batteries for Meta's AI glasses. It describes the shift from traditional wound electrode architectures to die-cut stacked layers to reduce impedance and enable higher peak power. It also highlights the importance of tight manufacturing tolerances for maximizing usable volume and energy density. The post further discusses the evolution of battery capacity and runtime across generations of smart glasses, including the system-level optimizations required to double runtime without changing battery chemistry. It also touches upon the complexities of dual-battery systems in Oakley Meta Vanguards and the power demands of the Meta Ray-Ban Display glasses.
  2. Meta 3D AssetGen: Generating 3D Worlds With AIThis post introduces AssetGen, a new foundation model for 3D assets, and discusses its development and training. It highlights the role of LLMs in generating entire 3D worlds from text prompts, contributing to Meta's ongoing efforts in AI-powered VR and AR development.
  3. Building multimodal AI for Ray-Ban Meta glassesThis post details the application of multimodal AI, specifically foundational models, to the Ray-Ban Meta glasses. It highlights the challenges of integrating AI into wearable devices and discusses the team's research in areas like AnyMAL, which processes text, audio, video, and IMU data. The post touches upon the request processing pipeline from the glasses, scaling considerations, and the potential for optimization.
  4. Meta Quest 2: Defense through offenseThis post details the proactive security research conducted by Meta's Native Assurance team on the Meta Quest 2's Android-based OS. It describes the discovery and exploitation of a memory corruption vulnerability in the VR Runtime service, which allowed for arbitrary native code execution from an unprivileged application. The post outlines the VROS architecture, the VR Runtime service, its attack surface, and the technical details of exploiting a specific vulnerability to hijack control flow, ultimately contributing to improved security posture for Meta Quest products.
  5. Meta’s head of AR hardware on the future of ARThis post introduces Caitlin Kalinowski, Meta's Head of AR Glasses Hardware, and her perspective on the future of AR hardware development. It highlights the significant challenge of creating AR hardware that is smaller, lighter, and as powerful as current VR headsets, emphasizing the 'zero-to-one' innovation required. The discussion touches upon the differences between VR and AR, the critical role of diversity in innovation, and the importance of fostering a safe environment for experimentation and failure in the development process.
  7. Inside Meta’s first smart glassesThis post details the engineering challenges and solutions involved in developing the Ray-Ban Stories smart glasses, focusing on integrating cameras, microphones, audio, and touch controls into a standard eyewear form factor. It highlights the need for a fashionable, comfortable, technologically advanced, lightweight design with long battery life, positioning these glasses as a stepping stone towards future AR glasses.
  8. Building a cross-platform runtime for ARThis post details the technical approach to building a cross-platform runtime for Meta's AR platform. The core innovation is the strategic decomposition of a monolithic runtime into smaller, modular plugins. This allows for dynamic inclusion or exclusion of AR capabilities based on device requirements and user configuration, enabling optimization for a wide spectrum of hardware, from high-end VR headsets to lower-end mobile devices. The focus on performance optimization and asset management is key to achieving broad accessibility and supporting diverse use cases with a single AR engine.
  9. Building lifelike virtual avatars to help people connect in VRThis post details the progress made by Facebook Reality Labs (FRL) in creating lifelike virtual avatars for VR. It highlights the completion of two capture facilities (face and body) designed for high-detail reconstruction and motion measurement. A key advancement is the development of an automated pipeline to create photorealistic avatars, enabled by recent breakthroughs in machine learning for real-time animation.
  10. Open-sourcing DeepFocus, an AI-powered system for more realistic VR imagesIntroduced DeepFocus, an AI-powered framework for real-time rendering of realistic retinal blur (defocused blur) in VR. This system utilizes an end-to-end convolutional neural network with novel volume-preserving interleaving layers to reduce spatial dimensions while preserving image details, enabling compute-intensive visual effects in real-time. The work is open-sourced along with a dataset to benefit the VR research community.
  11. Oculus Connect 5: Tech talks roundupThis post summarizes technical talks from Oculus Connect 5, detailing advancements in VR development. Key contributions include: bootstrapping social VR with Unity (invites, matchmaking, VoIP, P2P networking), advanced rendering with Unity's SRP (High Definition Render Pipeline, frame rate, resolution, cross-screen consistency), performance optimization for Oculus Rift and Quest using RenderDoc (texture/geometry limiting), deep learning for interactive actors in VR (obstacle avoidance, reactive motion), Oculus Lipsync SDK 1.30 (cross-platform integration, online/offline animation, laughter detection, DSP acceleration), sound design for Oculus Quest (handling multiple sounds, spatial audio mapping), and Oculus Rift PC SDK advancements (adaptive compositor kickoff, high precision dithering, Asynchronous Spacewarp).
  12. React 360 replaces React VR for streamlined development focusThis post announces the rebranding of React VR to React 360, reflecting its broader use cases beyond VR headsets, including interactive tours, brand promotions, and interactive news features accessible in any WebVR-enabled browser. Key improvements in React 360 include enhanced 2D interface rendering in 3D space, improved support for 180° mono and stereo video with built-in transitions and faster loading, and significant performance enhancements, particularly on lower-end mobile devices, due to runtime architecture changes.
  13. 2017 Year in review: Building immersive experiencesThis post details advancements in AI for immersive experiences in 2017. Key contributions include: 1. Evolving translation models from phrase-based to neural networks, powering over 4.5 billion daily translations. 2. Applying computer vision for automatic alt text on photos and improving image search. 3. Developing AR experiences using SLAM and smartphone cameras for overlaying digital art. 4. Applying style transfer to 4K 360 video for VR films, requiring a new approach due to computational demands. 5. Implementing end-to-end optimizations for 360 video streaming, including offset cube maps and view prediction models, leading to significant bitrate reduction and reduced buffering. 6. Announcing a spatial audio system supporting both spatial and head-locked audio simultaneously for 360 video.
  14. SLAM: Bringing art to life through technologyThis post details the engineering challenges and solutions involved in deploying Simultaneous Localization and Mapping (SLAM) for mobile Augmented Reality (AR) at Facebook. Key contributions include tailoring SLAM algorithms for diverse mobile devices with unique calibration requirements (focal length, distortion, rolling shutter, IMU synchronization), optimizing the SLAM library size from 40MB to under 1MB by refactoring and using common Facebook libraries, and developing the WorldTracker API to combine SLAM with other tracking algorithms for a more robust and believable AR experience. It also highlights the process of integrating artist workflows into AR development, demonstrating the practical application of SLAM for creating geolocated and persistent AR art installations.
  15. Using AI for new visual storytelling techniques in VRThis post details the application of AI-driven style transfer techniques to generate special effects for VR films, specifically for the "Alteration" VR experience. It addresses the technical challenges of applying style transfer to high-definition, stereoscopic, and 360-degree VR video, including memory and time constraints, and the subjective challenge of optimizing AI models against aesthetic opinions. The post describes retraining neural networks with larger image resolutions (768x768) for style transfer, the computational cost (120 GPU-hours for training, 550 CPU-hours for encoding), and the method for ensuring stereoscopic consistency by mapping style transfer from the left eye to the right eye using disparity maps. This demonstrates AI's maturity as a creative tool for filmmakers in VR.
  16. Surround 360: Beyond Stereo 360 CamerasThis post details the engineering challenges and solutions behind Meta's Surround 360 camera system, which goes beyond stereo 360 cameras to capture and process immersive video. It discusses the hardware design, including the arrangement of 17 cameras to capture a full spherical view, and the software pipeline for stitching, depth estimation, and rendering. The system addresses the complexities of handling high-resolution spherical video, enabling more realistic VR content creation.
  17. Announcing 360 Capture SDKIntroduced the 360 Capture SDK, enabling instant capture of PC VR experiences as 360 photos and videos. This SDK leverages cube mapping instead of traditional stitching, resulting in improved accessibility (works on baseline VR hardware), higher quality (captures at resolutions up to 4K), and faster capture speeds (under one second for VR-quality 360 video at 30 fps while maintaining 90 fps for the VR experience). The SDK is compatible with Unity, Unreal, and native game engines, and supports NVIDIA and AMD GPUs.
  18. Facebook open source at F8 2017This post announces the open-sourcing of React VR, a new library that allows developers to build virtual reality experiences on the web using JavaScript and standard web tools, expanding on the declarative programming style of React and React Native.
  19. Building virtual reality experiences on the web with React VRThis post introduces React VR, a new library that allows web developers to build virtual reality (VR) experiences using JavaScript and standard web tools. It leverages React's declarative programming style and extends it to 3D scene composition, integrating with WebGL and WebVR for headset compatibility. The architecture follows React Native's pattern of separating React reconciliation and rendering into different contexts using web workers for asynchronous communication, ensuring high frame rates for immersive experiences. React VR is built as a true React Native platform, allowing for code reuse and familiar components like View and Text.
  20. End-to-end optimizations for dynamic streamingThis post details end-to-end optimizations for dynamic streaming of 360 video, building upon previous work. Key contributions include the introduction of variable offset cubemaps for encoding, which allows for more efficient pixel allocation towards the viewport, reducing overall resolution without sacrificing perceived quality. It also describes improvements in stream selection logic by considering six factors (resolution, video quality, current view orientation, stream viewport direction, length of the offset, and field of view) to decrease interruptions by over 80%. Further optimizations include more frequent bandwidth estimation, smarter handling of video chunks (playing fetching chunks only when no fallback is available, and canceling replacement chunks if a fallback is used), and prefetching initialization and index chunks for DASH streams. Player performance was boosted by filtering stream candidates early, pre-computing intensive computations, and using lookup tables.
  21. Lessons from building social experiences in VRThis post details early explorations into social VR experiences, focusing on creating expressive avatars that enhance social presence through realistic yet not uncanny designs, incorporating arms, body approximations, realistic eye movements, and voice-triggered mouth animations. It also explores interactive objects and the ability to share 2D, 360 photos, videos, and Facebook posts within VR, as well as enabling real-time sharing of VR experiences with non-VR users via Messenger video calls.
  22. Gear VR to get Dynamic Streaming for 360 videoIntroduced dynamic streaming technology for 360 video to Gear VR, enabling more efficient delivery of VR video content. This technology aims to make videos look clearer and play faster for users.
  23. Oculus scientist wins computer vision awardThis post announces that Oculus scientist Richard Newcombe and partners won the Best Paper Award at CVPR for their paper 'DynamicFusion: Reconstruction and Tracking of Non-rigid Scenes in Real-Time'. The paper introduces the first real-time dense dynamic scene reconstruction system, which allows for the scanning of moving people and objects, enabling new applications in virtual and augmented reality.