Meta Reveals VR Headset Prototypes Designed to Make VR ‘Indistinguishable From Reality’

43

Meta says its ultimate goal with its VR hardware is to make a comfortable, compact headset with visual finality that’s ‘indistinguishable from reality’. Today the company revealed its latest VR headset prototypes which it says represent steps toward that goal.

Meta has made it no secret that it’s dumping tens of billions of dollars in its XR efforts, much of which is going to long-term R&D through its Reality Labs Research division. Apparently in an effort to shine a bit of light onto what that money is actually accomplishing, the company invited a group of press to sit down for a look at its latest accomplishments in VR hardware R&D.

Reaching the Bar

To start, Meta CEO Mark Zuckerberg spoke alongside Reality Labs Chief Scientist Michael Abrash to explain that the company’s ultimate goal is to build VR hardware that meets all the visual requirements to be accepted as “real” by your visual system.

VR headsets today are impressively immersive, but there’s still no question that what you’re looking at is, well… virtual.

Inside of Meta’s Reality Labs Research division, the company uses the term ‘visual Turing Test’ to represent the bar that needs to be met to convince your visual system that what’s inside the headset is actually real. The concept is borrowed from a similar concept which denotes the point at which a human can tell the difference between another human and an artificial intelligence.

For a headset to completely convince your visual system that what’s inside the headset is actually real, Meta says you need a headset that can pass that “visual Turing Test.”

Four Challenges

Zuckerberg and Abrash outlined what they see as four key visual challenges that VR headsets need to solve before the visual Turing Test can be passed: varifocal, distortion, retina resolution, and HDR.

Briefly, here’s what those mean:

  • Varifocal: the ability to focus on arbitrary depths of the virtual scene, with both essential focus functions of the eyes (vergence and accommodation)
  • Distortion: lenses inherently distort the light that passes through them, often creating artifacts like color separation and pupil swim that make the existence of the lens obvious.
  • Retina resolution: having enough resolution in the display to meet or exceed the resolving power of the human eye, such that there’s no evidence of underlying pixels
  • HDR: also known as high dynamic range, which describes the range of darkness and brightness that we experience in the real world (which almost no display today can properly emulate).

The Display Systems Research team at Reality Labs has built prototypes that function as proof-of-concepts for potential solutions to these challenges.

Varifocal

Image courtesy Meta

To address varifocal, the team developed a series of prototypes which it called ‘Half Dome’. In that series the company first explored a varifocal design which used a mechanically moving display to change the distance between the display and the lens, thus changing the focal depth of the image. Later the team moved to a solid-state electronic system which resulted in varifocal optics that were significantly more compact, reliable, and silent. We’ve covered the Half Dome prototypes in greater detail here if you want to know more.

Virtual Reality… For Lenses

As for distortion, Abrash explained that experimenting with lens designs and distortion-correction algorithms that are specific to those lens designs is a cumbersome process. Novel lenses can’t be made quickly, he said, and once they are made they still need to be carefully integrated into a headset.

To allow the Display Systems Research team to work more quickly on the issue, the team built a ‘distortion simulator’, which actually emulates a VR headset using a 3DTV, and simulates lenses (and their corresponding distortion-correction algorithms) in-software.

Image courtesy Meta

Doing so has allowed the team to iterate on the problem more quickly, wherein the key challenge is to dynamically correct lens distortions as the eye moves, rather than merely correcting for what is seen when the eye is looking in the immediate center of the lens.

Retina Resolution

Image courtesy Meta

On the retina resolution front, Meta revealed a previously unseen headset prototype called Butterscotch, which the company says achieves a retina resolution of 60 pixels per degree, allowing for 20/20 vision. To do so, they used extremely pixel-dense displays and reduced the field-of-view—in order to concentrate the pixels over a smaller area—to about half the size of Quest 2. The company says it also developed a “hybrid lens” that would “fully resolve” the increased resolution, and it shared through-the-lens comparisons between the original Rift, Quest 2, and the Butterscotch prototype.

Image courtesy Meta

While there are already headsets out there today that offer retina resolution—like Varjo’s VR-3 headset—only a small area in the middle of the view (27° × 27°) hits the 60 PPD mark… anything outside of that area drops to 30 PPD or lower. Ostensibly Meta’s Butterscotch prototype has 60 PPD across its entirely of the field-of-view, though the company didn’t explain to what extent resolution is reduced toward the edges of the lens.

Continue on Page 2: High Dynamic Range, Downsizing »

1
2
3

This article may contain affiliate links. If you click an affiliate link and buy a product we may receive a small commission which helps support the publication. See here for more information.


  • Engorged

    Wow, amazing stuff coming from Meta. They really see the full potential of VR.

  • Steven Lewis

    Thanks for this very detailed article! Fascinating stuff.

    • Jasmine Scott

      I am able to get paid 80 dollars hourly to do easy tasks on home computer.^gs28^I have not ever thought like it could be even possible but one of my confidant mate was collecting $25k just within four weeks completing this easy offer and also she has influenced me to join.^gs28^Get further details visiting this page….
      —->>> https://g­o­.­sv/hy5qdS

  • XRC

    Great article and good information dump from meta. Fascinating to see what some of those billions have bought.

  • xyzs

    Meta means beyond in ancient greek..?
    They should call their company MetaLate.

    • Angelos Panayiotou

      Just means later or after in modern Greek

  • Martijn Valk

    Very informative article, thanks! Still not a fan of Meta but I have to admit it’s impressive what they’re achieving so far.

  • Bram

    holocake 2 looks very familiar…a while ago we have seen a leaked picture by meta themselves of Andrew Bosworth wearing it and the media picked it up as the upcoming project Cambria, but now we know it wasn’t that. absolutely interesting design, i think it’s what we imagine a vr headset will look like at some point. Still, no word about fov in this context, unfortunately…

    • dk

      that headset is basically a test vehicle for the optics …other aspects r not developed furter in that test headset ….Norm said its basically a quest but tiny and it’s also wired to a pc it’s not standalone
      …there r 2 long videos about the headsets …one from meta themselves it’s on uploadvr’s youtube and the orher is on tested youtube channel

  • James Hughes

    Our society, government, planet-environment and interpersonal relationships have gone to shit. Instead of people using virtual reality equipment to look around the artificial world, move around in it, and interact with virtual features or items, why don’t we throw billions at solving real-world problems and have real world interactions?

    • Walkabout

      Sometimes solutions come from indirect technologies. You can definitely cut down on emissions by not needing to travel as much if VR gets good enough for example.

    • dk

      why r u wasting time writing a comment when in that time u could have fed a homeless kitten …what a monster

      if billions weren’t spent on various industries u wouldn’t be able to say whatever u want to anyone around the planet interested in this article

      ….u use headphones right ….arvr will augment your vision the same way headphones augment your hearing……and one day u will be able to meet any relative or anyone around the world indistinguishable from meeting in reality

      • James Hughes

        Why aren’t you drinking semen instead of wasting my time.

        • dk

          xD haaaa well how do u know I’m not doing that in my free time
          ….it’s funny where your mind went when u realized I’m right

      • I wanna meet a virtual Moe Szyslak.

    • Rand

      Sure, we are a society that blames the tools for the crime. Guns kill people, spoons make you fat, phones and tech make people less human. No on all counts. You know what makes young people have issues meeting people? The fact that mommy and daddy tossed a tablet in front of their kid and let it take over parenting. Kids bury their faces in phones and tablets to avoid social situations. Good parents don’t let that happen. Want a kid that will rule the world? Let them learn the tech but do not let it own them, do not use it as a crutch.

      Kick your kid out of the house without a phone, its okay, we survived as a species without a digital tether for hundreds of thousands of years. Tell them not to come back till the street lights turn on… but when they do they have 15 min. or else. Then let them have the phone. Its okay, they can enjoy technology.

      VR is not a problem for normal people…. the problem is there are less and less normal people.

    • kontis

      WHY ARE YOU EXPLORING THE ISLAND INSTEAD OF FIGHTING THE MOLD IN OUR CAVE?! OUR KIDS ARE DYING AND YOU DO STUPID THINGS LIKE GOING OUTSIDE.

      What curiosity ever gave us? 100x reduction in poverty, 10x reduction of child mortality and 3x longer living. For what? To suffer more More people can live now so mor people suffer in “society” and they suffer for longer because they live longer.
      AMEN Brother! We are the cancer of this world and this is really BAD!

  • Foreign Devil

    That high nits headset needed high level cooling with those bright lights. You could probably blind yourself looking at the VR sun in a game using that one. I’m most looking forward to 20/20 vision resolution while maintaining high FOV.

    • dk

      the percentage of light getting through is not high it’s matching indoor lighting still nowhere close to light outside

    • Well, with a virtual Sol being just like the real one,
      as they also tell you IRL: DON’T STARE AT THE SUN.
      There.
      See how easy that is …?

    • overzeetop

      You can miss me with HDR. Since black (with an oled or similar) is actually 0 light, the only way to get higher dynamic range is to increase the brightness. It’s like offering a headphone that can produce 200dB of sound you really feel like you’re in the concussion zone of a munition. I don’t want to suddenly be blinded by a scene, going from dark to full daylight. without some kind of safety interlock, this feels like a lawsuit waiting to happen. And with an interlock, it’s still a lawsuit, just with a longer fuse until the interlock is broken or circumvented.

  • Till Eulenspiegel

    One day you will tell your grand kids that people used to wear a box on their face. It’s like seeing those photos of luggage bag size laptop. People will also laugh at how people had to turn this round thingy to drive a car.

    • kontis

      Just like our grandparents told us how they had to use big, bulky headphones to have proper acoustics and ear isolation and now we have that perfect sound in tiny earplugs… oh, wait no we don’t. W still buy bulky headphones to achieve that.

      Or like our grandparents had to buy big cameras with big lenses to get great bokeh and a very high amount of light to make really beautiful photos. Now we can just pull out a smartphone and we get the same bokeh…
      Oh, wait, no we don’t. It’s just a post processing filter that never works correctly and the amount of photons hitting the tiny lens is never as good. So photographers still use cameras as bulky as 50 years ago.

      Or how people in the early 00s had tiny cellphones that fit even in tiny pockets and anyone using large cellphone was lame for having a big brick and now… everyone has a brick. Wait… wasn’t the tech supposed to miniaturize?

      Or how our grandparents had watches with batteries that lasted decades. And now the battery lasts 24 hours. Such progress… :)

      The point is: technology isn’t magic. Physics are a serious limitations. Some things will never be miniaturized. It never goes exactly as predicted. Same will probably happen to XR.

      • Till Eulenspiegel

        You can continue to use this IBM hard drive, it’s as big as a fridge and has a huge 5MB storage.

        https://uploads.disquscdn.com/images/ddacb802f0f290343bb1f59700e40d3cdc46d8a3044f0c55bac6d7b07426d1bc.jpg

      • overzeetop

        Ok, boomer.

        Seriously – practically every case you mention relies on a standard that is the realm of the 0.001%. Headphones for great acoustics – yes, IEMs produce nearly everything – *and* add noise cancelling. Computational photography? Modern smartphones produce results in low light which was impossible with film and even an F1.0 lens of 40 years ago. You can still get more with optics – but 99.999% of the time its just not necessary. And, as for bokeh, if you live in a world where it’s required and the automation isn’t sufficient, we can add it in post to look better than – or exactly like, any glass optics. Brick smartphone? Try miniaturized computer, camera, and GPS tracker that can place a video call. If you want a little phone, connected devices allow you to talk to anyone in the world with a mere pendant or a watch. Watch batteries lasted decades? Oh, sure, just as long as you reset them manually to make up for inaccurate time keeping; todays connected devices are synchronized with atomic clocks. BTW – did your grandparents own a watch that could make a phone call, check the weather anywhere in the world, give you turn by turn, real-time directions, call a family member if you fall, or detect a cardiac arrhythmia? No, I didn’t think so.

        Physics will limit certain functions, but we’re learning new tricks to compensate for those limitations on a daily basis. What our grandparents did with the tools they had was phenomenal, but no more so than the true cutting edge of development today.

  • Very very interesting! I mean, it’s all smoke, because nothing of this is coming to our shelves soon, but it’s interesting to have a peek into what they are working on

  • alxslr

    So FOV is way down on the list, after varifocal, HDR, retina res, and almost everything else.
    Help us Valve/Sony, you’re our only hope.

    Great for the rest of innovations and ideas, but this absence of interest in field of view is so sad…

    • Rainy

      FOV is a distortion problem that is being worked. it’s not as simple as bigger lenses or simply willing it to be so, as the lenses have to be designed with the desired FOV in mind and software has to correct that lenses distortion, and the best way to do that is to have a camera track the eye and do software adjustments. The reason most high-fov headsets today are Fresnel is because of their ability to be stacked, but these headsets also have the highest levels of distortion (pimax, vive pro 2). The demonstration on distortion was related to FOV as we need to find a solution for the distortion before we can raise the FOV, but you have to go one step at a time. It also doesn’t make sense to research only one thing at a time, all these headsets were primary separate from the others and were used to prototype specific technologies, not a whole stack. Their current prototypes are low FOV because they are focusing on the effects of clarity, and introducing extra variables like the distortion (which is a separate research project in itself) could skew the results. The FOV “prototype” (not really, more of the research branch for that) can be considered the TV that simulates distortion in lens designs. Eventually the tech stack will find a spot to merge together.

      • kontis

        One challenge is optical (physics), another one is rendering (software and computing hardware). Both are HUGE and “not worth it” for companies targeting mainstream, especially because wider FOV makes resolution and the sim sickness problems even worse. So NOT bothering with FOV helps them in 4 different ways. The incentive of the market is insanely anti-FOV currently…

        The incredible irony here is the fact that current XR revival was caused by a HMD that gave up on everything to achieve wide FOV (Oculus DK1 – it had wider FOV than anything Oculus ever released again…)

  • jimbo5Z

    Over an hour hands-on with Meta’s New VR Headset Prototypes. On Adam Savage’s Tested with Norman Chan – on Youtube

  • philingreat

    That doesn’t make any sense. Why would Meta announce that before releasing the Cambria?

    • Usually, companies release news like Meta did yesterday whenever
      they get wind of a competitor’s about to make an important announcement.

      • philingreat

        True, but those rumors are going on for years, and according to those rumors, the cambria will release before the competitor, so they could’ve also just waited until after the release of cambria

    • dk

      it’s an ad ….explaining to the stock investors where the billions for rnd r going

      • benz145

        This does seem like the most likely motivation.

      • philingreat

        unfortunately, the stock market doesn’t care about VR. Facebook looses 0.5% of users and their stock tanks. The Quest sale numbers can grow 100% and no one cares. This announcement had nearly no impact on the stock market.

        • dk

          a public company has to explain to the shareholders their 10 billion a year spending on ar/vr …especially if he wants to keep doing that and not get sued

  • xyzs

    They could also do some minor hardware upgrades but more often instead of waiting 10 years that all their issues are fixed… A Quest 2 Pro with a better SoC and better screens wouldn’t hurt. Cool to see that but for us who are not working in their labs, that just doesn’t exist.

  • 144Hz

    It’s nice to see that Zuck is really into vr and he’s taking it seriously. I’ve never seen him excited about anything before.

  • This is everything we already know about, just presented rapidly and with less information by Mark Zuckerberg himself.

  • Mateusz Pawluczuk

    Fascinating, HDR sure is the most challenging of all but is Abrash right saying it’s the most important? Human eye resolution, human eye fov, varifocal seem quite important imho

    • benz145

      You might be surprised how important brightness is to human perception. Even just changing the brightness on your smartphone can make it easier to see details and drastically changes the perception of colors. It’s almost impossible to find a normal scene in RL that doesn’t have elements of contrast that fall into the HDR range—even indoors. That level of contrast has a certain ‘feel’ that just isn’t represented on non-HDR displays. That ‘feel’ contributes a lot to what it means to be looking at reality.