Apple and Facebook parent Meta are expected to release mixed reality headsets in the coming year that could finally fulfill the industry’s promise to turn head-worn devices into the next big shift in personal computing.
But there’s one major potential snag: sticker shock.
The best-selling virtual reality headset, the Meta Quest 2, retails for $400 and accounted for 78% of the nascent VR market in 2021, according to IDC. Consumers who want the next-generation technology are going to have to spend multiples of that.
Meta’s forthcoming high-end headset, codenamed Cambria, is expected to cost at least $800, the company said earlier this year. Apple’s unannounced device could reportedly cost thousands of dollars. That’s a hefty load for products in a category that’s yet to go mainstream. Just 11.2 million VR units were shipped last year, IDC said. Apple sells that many iPhones every few weeks.
To expand the market, Meta and Apple will have to convince consumers that more advanced systems will be worth the investment. Both companies are reportedly betting on a new technology called passthrough mixed reality, which requires better displays and more processing power.
If passthrough mixed reality works as advertised, a VR headset would also function as a set of augmented reality glasses, enhancing the possibilities for applications and real-world use.
With existing VR devices, the experience is limited to what’s on the headset’s display. In passthrough AR, powerful cameras on the outside of a VR headset take video of the outside world and send it to two or more displays, one each in front of the user’s eyes.
This allows for developers to play with mixed reality, overlaying software or graphics on the video of the real world from just outside.
Believers in mixed reality say that we’ll eventually be able to condense the technology into a lightweight pair of glasses with transparent lenses. But that’s for the future.
The passthrough approach is emerging as the preferred near-term option because optical transparent displays are nowhere near ready for primetime. The problem for today is that passthrough mixed reality requires a lot of expensive parts and a powerful headset, limiting the size of the market.
In addition to the advanced cameras, passthrough devices need depth sensors that can take detailed video and measurements of the user’s surroundings. They also have to track the user’s eyes so as not to waste power generating graphics that will go unseen. And they need powerful processing capabilities and software to reduce latency so that what the user sees inside the headset isn’t delayed or blurred.
Most important is the high-resolution screen that needs to be much denser than a smartphone display because it’s so close to the user’s eyes. Smartphone screens average about 550 pixels per inch, but mixed reality devices require displays with about 3,500 PPI, according to CounterPoint Research.
While Meta and Apple haven’t released their headsets, a few devices currently on the market support passthrough mixed reality. The experiences tend to be limited — black and white or low-quality video — because of a lack of processing power.
A few weeks ago, I was able to test a headset from Varjo, a Finnish company co-founded by Urho Konttori, a former Microsoft and Nokia executive. Last year, Varjo released the XR-3, which offers full-color, low-latency passthrough mixed reality. It’s expensive, heavy, and aimed at businesses. It costs $6,495 to purchase or about $1,500 to rent it for a year.
In playing around with the XR-3, I felt less isolated than with other VR headsets.
I could access a virtual world with the press of a single button, and I could pull up games that took over my entire field of view. I could use virtual computer monitors displaying Windows applications inside the virtual world.
I was also able to interact with the world around me through Varjo’s passthrough view. In the demo, Varjo placed a life-size car model inside the space. I was able to walk around it and inspect its interior and discuss what I was seeing with someone who wasn’t wearing a VR headset.
Most impressively, when passthrough was turned on, I could interact with the actual environment around me, carrying on a conversation with the person next to me or finding a chair and sitting in it. This isn’t possible with existing VR technology, which forces you to remove yourself from the physical world.
Konttori told me that was one of his main goals. The company wants to almost mimic “human-eye” display quality, which he calls the “holy grail” of mixed reality.
‘A single coherent scene’
The XR-3 has two 2880 by 2720 pixel displays, and the company uses eye tracking to focus its processing power to deliver better image quality where your eyes are looking.
The key is “being able to merge the physical reality around you with the virtual reality objects and make it into a single coherent scene, where you cannot tell apart what is real and what is virtual anymore,” Konttori said. “Part of this evolution is that you can see that at some point, the fidelity of this experience is equal to what you would perceive by looking at it with just your own eyes.”
However, to use the XR-3 you have to be tethered by a cable to a powerful gaming PC. Meta and Apple are focused on developing devices that don’t require attachment to a separate computer. Konttori knows it will be hard for his startup to compete with some of the biggest tech companies in the world, but he says Meta and Apple still face challenges.
That’s because developing a consumer-friendly product with the right weight and power consumption is very tricky, especially when it comes to keeping costs down and shipping millions of them.
“Companies are focusing on consumer-alike experiences, which means that they are still really driven by the size, weight, ergonomics point of view, as well as cost,” Konttori said.
Apple is notoriously secretive about its product roadmap, especially when it comes to new categories. The company has invested heavily in virtual reality research and development in its Technology Development Group and has purchased several startups specializing in mixed reality technology.
According to reports from Bloomberg and The Information, Apple is developing a mixed reality headset that resembles ski goggles with a powerful homegrown chip, similar to what powers its MacBook laptops, and higher-resolution displays than what’s currently on the market.
The headset will reportedly support passthrough video and offer games and other applications. At one point, Apple was aiming for at least resolution similar to a 4K TV per eye for its first headset, because anything less could result in users seeing individual pixels, The Information reported.
Apple hasn’t confirmed its plans to release a mixed reality headset, and the company didn’t respond to a request for comment on this story. In an interview with Chinese media earlier this year, Apple CEO Tim Cook suggested that something is in the works.
Meta has said Project Cambria, with support of color passthrough, is scheduled to be released later this year. Based on renderings of the device that have been made public, it also looks like a pair of ski goggles. It will include pancake optics, a type of lens that doesn’t need to be calibrated as finely as other VR lenses.
Meta said in May that the price for Cambria would be “significantly higher” than $800.
While passthrough technology has yet to hit the market in a real way and will be quite pricey once it does, metaverse developers are rallying behind it. The primary alternative, optically-based mixed reality, uses transparent displays built into lenses to integrate computer graphics with the real world. Microsoft’s Hololens and Magic Leap use optical waveguides, a type of transparent display.
Transparent displays are also expensive, and they have their own sets of challenges. They’re not good when used in bright daylight, and the current offerings can suffer from poor image quality and blurry text.
Varjo is making a bet on passthrough technology and Konttori says it’s the better approach in large part because it’s completely digital, putting more control in the hands of developers.
“It becomes computable,” Konttori said. “It becomes a tool for artificial intelligence to be participating in your world, enhancing your view or your intellect, and you can distort the world in the tiniest ways or the biggest ways possible.”
He expects passthrough to be “the winning approach for a very, very long time.”
WATCH: The future of entertainment is mixed reality gaming experiences