BMW and Meta’s Reality Labs Research have made significant strides in the realm of virtual reality (VR) and more importantly mixed reality (MR) in digital vehicle ecosystems. The work paves the way for the future mixed reality experiences that BMW plans to put into production in just a few years.

The BMW Group Technology Office USA in Mountain View recently presented a research update, unveiling their groundbreaking work displaying stable VR and MR content to passengers while the vehicle is in motion, even during turns, speed bumps, and acceleration. This breakthrough was made possible through the integration of Meta Quest’s tracking system with BMW’s sensor data, enabling the creation of stable and immersive “car-locked” gaming, entertainment, productivity, and meditation experiences.

According to Claus Dorrer, the Head of BMW Group Technology Office USA, the full extent of this technology’s integration into customer products is yet to be determined. However, he envisions a myriad of potential use cases for extended reality (XR) devices in vehicles. These applications range from assisting drivers in locating their cars in crowded parking lots to providing real-time alerts about road hazards and offering crucial information about the vehicle’s condition. The implications of future augmented reality (AR) glasses and VR devices are vast, holding promise for both passengers and drivers. Through their research partnership with Meta, BMW aims to explore the possibilities of immersive in-car XR experiences, paving the way for the seamless integration of such devices into automobiles.

The proof-of-concept prototype developed by BMW and Meta addresses key technical challenges. Traditional VR headsets rely on cameras and motion sensors to track the user’s movements, ensuring that virtual content remains stable even as the user shifts their gaze or moves their head. However, within fast-moving vehicles, the headset’s sensors become confused due to conflicting signals from the device’s movement within the car and the vehicle’s movement in the external world. In essence, the headset’s motion sensors perceive motion, while its cameras perceive stillness when observing the car’s interior.

To overcome this challenge, researchers from BMW and Meta collaborated to incorporate information from BMW cars’ sensor arrays into the tracking system of the Meta Quest Pro. This integration enables the tracking system to simultaneously calculate the device’s location relative to the car and the car’s location relative to the external world. As a result, virtual objects can be accurately anchored to the moving car, allowing for captivating VR and MR passenger experiences.

Richard Newcombe, the Vice President of Research Science at Reality Labs Research, emphasizes the potential of this research prototype to revolutionize safe interactions with the environment during travel. Passengers can enjoy engaging and comfortable experiences that are aligned with the car itself, encompassing VR and MR gaming, entertainment, productivity, and even meditation. Looking ahead, as the technology progresses toward AR glasses capable of rendering reliable world-locked content, passengers may have the ability to visualize markers for landmarks, restaurants, and places of interest, among other features.

By merging the movement data generated by the vehicle with the sensor capabilities of XR headsets, it became possible to eliminate the effects of the vehicle’s motion and accurately track the headset’s movements within the car. This breakthrough facilitates the rendering of content that synchronizes seamlessly with the vehicle’s movement in real-time, whether inside or outside the car, even at high speeds and regardless of the road conditions.

The BMW Group has a track record of pioneering digital technologies such as the Digital Key, which enables vehicle access, and driving industry standardization for such innovations. The company’s network of technology offices worldwide has played a vital role in initiating groundbreaking projects like this one. In their quest to establish an industry standard for connecting XR consumer electronics devices to vehicles, BMW is open to collaborating with additional partners. Additionally, the company remains committed to the principle of privacy by design, ensuring that customer data is protected and securely processed in accordance with data privacy requirements, across all the markets it operates in.

For more information on the project can be found here.