Augmented Reality Will Drive the Next Generation of Infotainment Systems

Apple files an AR automotive patent application, while Mercedes-Benz is set to debut AR on its 2019 A-Class model.


By Murray Slovick, Contributing Editor

As a member of the fourth estate, I regularly receive lists of published technical articles and patent filing advisories. I’ll admit that with regard to the latter, I usually glance through them with something less than the rigor a horse-racing enthusiast perched at the rail might employ examining a copy of the Daily Racing Form.

Nevertheless, an Apple patent application published by the U.S. Patent and Trademark Office recently caught my attention. It shows the company’s interest in automotive tech goes beyond ride sharing and the 45 Apple vehicles California’s Department of Motor Vehicles says are being tested in that state.

Application No. 15/713,274, titled “Augmented Reality Display” delineates how Apple is considering ways to use augmented reality (AR) to create and display information about the road ahead, including objects that are out of the driver’s vision. The patent application was filed September 22, 2017 and published March 29, 2018.

For the uninitiated, AR is a technology that provides a real-time view of the physical world, where elements are augmented by computer-generated input such as sound, video, or graphics.

The AR Effect on Safety

Growing awareness of safety, security, and driver-assistance features, along with an increasing role of smartphones in the day-to-day life of individuals, is largely fueling the global auto infotainment market. However, driver distraction caused by interaction with infotainment systems could hamper growth in a market that’s 2018-2022 forecast “Automotive Connected Infotainment System Market Report” says will expand at a CAGR of 11.82%.

Instead of distracting the driver, the use of AR in navigation could improve safety as infrastructure-connected traffic information is placed right on the windshield. As for entertainment, head-up displays (HUDs) will be replaced by AR-enabled displays at windows or on seat backs, which will provide passengers with a more immersive experience.

And when autonomous vehicles finally arrive, AR will also help former self-drivers ease into their transition as passengers when getting behind the wheel. It will do so by providing a means of showing them what the self-driving car sees on the road.

The Apple patent application is for an “adaptive vehicle augmented reality display using stereographic imagery.” It proposes to combine data from the vehicle with a pre-generated stereographic reconstruction of the world created using real-world images collected from a large number of sources over time, “to improve rendering of 3D graphics content for AR views of a scene.”

The patent further states: “By leveraging the pre-generated 3D model, the AR system may use a variety of techniques to enhance the rendering capabilities of the system.” The AR system proposed by Apple may obtain pre-generated 3D tiles created from aerial or street photography, including data about road routes, points of interest, and other areas useful to drivers from a remote source (e.g., cloud-based storage), “and may use this pre-generated 3D data, for example, a combination of 3D mesh, textures, and other geometry information, to augment local data collected by vehicle sensors to determine information about a scene, including information about occluded or distant regions of the scene.”

Figure 1: The Apple patent application, reproduced here, shows how an AR display could provide information on things of interest that are out of view, such as the path that the road takes behind a hill or some trees. The three-digit numbers in the image refer to specific claims made in the patent application.

For instance, if a car in front of the AR-equipped vehicle drives out of immediate view, such as around a bend, a downloaded data tile could overlay a virtual car graphic on the display that shows where the vehicle might have traveled.  Prompts could be overlaid on the driver’s view of the actual road rather than via a map depiction.

As an example of how 3D AR would employ local sensor data, the patent application says the AR system “may utiize remote sensing devices to provide depth information about objects in the environment. In some scenarios, laser-based sensing technologies such as LiDAR can provide high resolution environmental data such as depth maps which may indicate the proximity of different objects to the LiDAR.”

Mercedes-Benz and AR

Apple isn’t alone in its pursuit of automotive AR. Daimler has stated it will debut AR navigation in its next-generation Mercedes-Benz A-Class vehicles as part of the Mercedes-Benz MBUX (Mercedes-Benz User Experience) in-vehicle infotainment system co-developed by Daimler and its tier 1 supplier Harman, a wholly-owned connected-car and audio technology subsidiary of Samsung Electronics Co. Ltd.

MBUX will provide turn-by-turn directions with AR visuals designed to keep drivers informed and feeling in-control. MBUX is built upon Nvidia’s GPU technology, and its capabilities are designed to grow over time. The new system will help OEMs prepare for Level 3, 4, and 5 autonomous vehicle operation. While in automated-driving mode, the AR-capable navigation system will show the driver what actions the car is performing in real-time, for example, when the car is about to make a turn, brake, or accelerate.

Figure 2

Figure 2: MBUX, the Mercedes-Benz User Experience, will enter series production this year in the new A‑Class.

MBUX also offers an in-car voice-activated assistant with natural speech recognition similar to systems like Amazon’s Alexa. The voice-powered assistant can be summoned by saying “Hey Mercedes” followed by your request, or by pressing a button on the steering wheel.

In a self-driving situation, the feature would eliminate the abstract map altogether to show occupants what the car is about to do next, such as make a turn, accelerate, brake, switch lanes, or merge. This intuitive navigation experience would be shown in real-life context, giving passengers a feeling of control despite the vehicle’s autonomous capabilities. By communicating these actions to occupants through intuitive visualizations, drivers will more comfortably transform into passengers through a better understanding of surrounding traffic and environmental conditions.

AR features aren’t exclusive to fully autonomous driving. The technology can also help drivers through advanced driver-assistance systems (ADAS) such as lane-departure warning, blind-spot monitoring, and automated cruise control.

Start typing and press Enter to search