Oculus Chief Scientist Dives Deep Into the Near Future of AR & VR

52

In his latest presentation at Oculus Connect 5, Oculus Chief Scientist Michael Abrash took a fresh look at the five-year VR technology predictions he made at OC3 in 2016. He believes his often-referenced key predictions are “pretty much on track,” albeit delayed by about a year, and that he “underestimated in some areas.”

Optics & Displays

Image courtesy Oculus

Revisiting each area of technology in turn, Abrash began by discussing optics and displays. His predictions for headset capabilities in the year 2021 were 4K × 4K resolution per-eye, a 140 degree field of view, and variable depth of focus.

“This is an area where I clearly undershot,” he said, noting that Oculus’ own Half Dome prototype shown earlier this year had already met two of these specifications (140 degree FOV and variable focus), and that display panels matching the predicted resolution have already been shown publicly.

SEE ALSO
Google & LG Detail Next-Gen 1,443 PPI OLED VR Display, 'especially ideal for standalone AR/VR'

Abrash highlighted the rapidly progressing area of research around varifocal displays, saying that they had made “significant progress in solving the problem” with the AI-driven renderer DeepFocus that can achieve “natural, gaze-contingent blur in real time,” and that they would be publishing their findings in the coming months.

Image courtesy Oculus

Beyond Half Dome, Abrash briefly mentioned two potential solutions for future optics: pancake lenses and waveguides. Like Fresels, the pancake lens isn’t a new innovation, but is “only now becoming truly practical.” By using polarization-based reflection to fold the optic path into a small space, Abrash says pancake lenses have the potential of reaching retinal resolution and a 200 degree field of view, but there would have to be a tradeoff between form-factor and field of view. Because of the way pancake lenses work “you can get either a very wide field of view or a compact headset […] but not both at the same time,” he said.

Image courtesy Oculus

But waveguides—a technology being accelerated by AR research and development—theoretically have no resolution or field of view limitations, and are only a few millimetres thick, and could eventually result in an incredibly lightweight headset at any desired field of view and at retina resolution (but that is still many years away).

Foveated Rendering

Moving on to graphics, Abrash’s key prediction in 2016 was that foveated rendering would be a core technology within five years. He extended his prediction by a year (saying that he now expects it will happen within four years from now), and that the rendering approach will likely be enhanced by deep learning. He showed an image with 95% of the pixels removed, with the distribution of remaining pixels dissipating away from the point of focus. The rest of the image was reconstructed efficiently through a deep learning algorithm, and it was impressively similar to the original full resolution version, ostensibly close enough to fool your peripheral vision. Foveated rendering ties closely with eye tracking, the technology that Abrash thought was the most risky of his predictions in 2016. Today, he is much more confident that solid eye tracking will be achieved (it is already part of the way there in Half Dome), but this prediction was also extended by a year.

Spatial Audio

Spatial audio was the next topic, and Abrash conceded that his prediction of personalised Head-Related Transfer Functions (the unique geometry of each person’s ear which influences how they perceive the soundfield around them) becoming a standard part of the home VR setup within five years might also need to be extended, but he described how a recent demo experience convinced him that “audio Presence is a real thing.” Clearly the technology already works, but the personalised HRTF used for this demonstration involved a 30-minute ear scan followed by “a lengthy simulation,” so it’s not yet suitable for a consumer-grade product.

Controllers & Input

Image courtesy Oculus

Regarding controllers, Abrash stood by his predictions of Touch-like controllers remaining the primary input device in the near future (alongside hand tracking). After running a short clip of one of Oculus’ haptic glove experiments, he adjusted his previous opinion that haptic feedback for hands wasn’t even on the distant horizon, saying that “we’ll have useful haptic hands in some form within ten years.”

SEE ALSO
Hands-on: HaptX Glove Delivers Impressively Detailed Micro-pneumatic Haptics, Force Feedback

Ergonomics & Form Factor

Abrash presented this sleek concept as plausible form-factor for an AR/VR headset once waveguide optics are mastered. | Image courtesy Oculus

On the subject of ergonomics, Abrash referred to the increasingly significant technology overlap between VR and AR research, noting that future VR headsets will not only be wireless, but could be made much lighter by using the two-part architecture already introduced on some AR devices, where heavy components such as the battery and compute hardware could be placed in a puck that goes in your pocket or on your waist. He said this companion device could also link wirelessly to the headset for complete freedom of motion.

Even still, optical limitations are largely the bottleneck keeping VR headsets from approaching a ski-goggle like design, but advances in pancake and waveguide optics could make for significantly more slender headsets.

SEE ALSO
Toward Truly Glasses-sized AR: First Look at DigiLens' AR HUD Reference Headset

Continued on Page 2 »

1
2

This article may contain affiliate links. If you click an affiliate link and buy a product we may receive a small commission which helps support the publication. See here for more information.


  • Thomas

    Man, that’s some awesome stuff!
    It’s mean, that you don’t tell which one of the last images is the real photo ;-)
    I’d say the left one is real, because the image is a bit sharper and there is a reflection on the tooth, that’s missing in the right image :-D

    • Foreign Devil

      I’m sure they are both photographs of a real person. . just one has the photo textures overlaid on a 3D mesh.

  • paratay

    Michael Abrash is a tool, ALL his predictions are obvious and he gets a big head about it. I can give you much more accurate prediction and I am prepared to bet him if he is man enough. He is wrong in most

    • Bob

      “For one thing how can an HMD like the Oculus Quest which is going backwards in Hardware performance be capable of the CG quality he is talking about.

      You truly are clueless aren’t you?

      • paratay

        What are you 8, dick head. yeah I clueee lessss. No you are correct, the piece of shit hardware on the Quest is a miracle. Sorry you win

        • saintkamus

          Don’t forget to breathe.

        • BananaBreadBoy

          Abrash’s talk was on *PCVR*, moron. Not Mobile headsets like Quest.

          • Kev

            Stuff that will not release for years, if ever. They have done nothing in the PC VR arena except create walled gardens within a budding industry that certainly doesn’t need massive infighting. It needs innovation (which they have done nothing) and hardware competition (also nothing). Best they can do for PC is release new versions of their software over and over and over.

        • Andrew Jakobs

          Maybe you should read what Abrash is talking about.. he’s not talking about the Quest..

        • IsaiahGilliland

          Better to remain silent and thought a fool than open your mouth and remove all doubt.

      • Engineer_92

        Not clueless, just an outright idiot

    • saintkamus

      You’re dumb.

    • Xron

      Well, maybe he is, maybe he is not, but for the most part, people care about what oculus can offer for them as 2nd gen pc vr (atleast me) and if they can offer something similar to Abrash’s predictions by 2020 summer, I would be really happy to buy it :)
      Commenters below, we’re all interested in Vr, don’t let us go down to flame war with each other like in some reddit forums.

    • MosBen

      The Quest won’t be as powerful as a 2080ti, but then, it doesn’t need to be. Future generations of the Quest will undoubtedly have more robust hardware, but it won’t ever match what a GPU in a big PC case is capable of doing. But again, it doesn’t need to be.

    • IsaiahGilliland

      He’s not talking to you idiot, he’s talking to the investors in the audience who barely know how to use email.

    • impurekind

      You know what would really help: You paying more attention to what is actually being said and shown and the context it was said and shown in. Just saying.

    • FloridaOJ

      Weak bait 2/10. The people who seriously responded to you.. they’re new to the internet and don’t count. Try harder next time.

  • Maciej Soinski

    The future is bright, but the deeper we go into development of this technology we are more aware that VR/AR needs not years but decades to mature. For consumers these are not good old days.

    • impurekind

      Most of the essentials for a genuinely great VR experience with little compromise will be nailed long before “decades”. It will just be real hardcore stuff that most consumers could essentially live without, like having full body tracking with haptic feedback and that kind of thing, that will take a lot longer. But I’d guess that withing a generation or two we’ll have affordable VR headsets with resolutions that are high enough that any further improvements will be more welcome than essential, a field of view that is pretty on-spec with the actual human field of view and therefor doesn’t need to be any higher, a refresh rate that is also high enough that anything beyond will similarly be welcome but not essential, fully 3D sound that is so good that anything else beyond it is barely even perceptible at that point, and so on. So, yeah, the [near] future of VR is looking very bright indeed.

  • Andrew Jakobs

    Problem with AR headsets is that they tend to be a screen in front of your vison of the real world (and that’s a good thing ofcourse), but VR IMHO requires a headset that will block all outside light/vision. So the demand of both headsets are quite different.

    And as always, the biggest problem at the moment isn’t really the headset itself, but the GPU’s, they just aren’t powerfull enough for mainstream adoption with AAA expected visuals. Even the GTX1060/6GB is still 250+ euro, and that’s about the lowend GPU you need for even a bit of VR with the current Rift/Vive, for higher resolutions you’ll need much better GPU’s, even the 2080ti isn’t capable of driving a 4k per eye display at 90+ fps with everything on expected visuals. And foveated rendering is nice and all, but as long as decend eye/gaze tracking isn’t available, it’s still a long way off..
    For the real enthousiast with enough money or Businesses it might be affordable to go for the highend VR, but for VR to be mainstream affordable, which means less then $400 ALL-IN with visuals we expect these days from AAA games, we’re still at least 4 years away..

    • MosBen

      I agree that $400 all-in is the sweet spot for mainstream VR, and I think that Oculus agrees with us, hence the Quest. Sure, in ten years the Quest’s hardware will be quaint, but for the purposes of next year, I think that it’s worth non-VR nerds getting excited about.

    • impurekind

      Well this is kinda the entire point of his presentation: He’s predicting where he thinks this stuff will be in the [near] future. And, literally, basically all his predictions were for about four years in the future.

      • daveinpublic

        Glad to know there are companies out there that are still investing everything they can into it.

  • Kev

    Oculus sure talks alot about things they can’t even demo. I bet they are the last to market with every one of the innovations he discussed.

    • BananaBreadBoy

      Nobody cares about what comes first, only what does it right. No one cared about the IBM Simon or LG Prada even though they came before the iPhone ¯_(ツ)_/¯

      • Kev

        They’ve been actually going backwards. Instead of showing evolving VR instead they went for battery operated smartphone devices with even lower specifications. Next year we can look forward to a three year old snapdragon 835 device from Oculus!! It will have the same 6DOF capability that the DK1 had in 2013! Sweet.

    • impurekind

      The stuff they talk about is often the same stuff they demo the next Connect and then tend to release the next one or two after that. That’s not too shabby at all considering they are both predicting and then then actually building the future of VR. And, on multiple occasions they’ve been the first to market with VR headsets that are beyond the rest of the competition. The Quest being a perfect example of this, being the most powerful fully wireless/untethered VR headset out there that also has full 6DoF on both the headset and controllers. So it’s both inaccurate and unfair to act like Oculus isn’t delivering the goods when it actually is–regularly.

      • Kev

        We had 6DOF in the DK1 in 2013. I would also point out the Quest can only track your controllers right in front of you. You cannot swing behind your head or as low as your beltline. It’s far more limited than even the existing touch controllers.

        • impurekind

          Not on an all-in-one completely wireless untethered headset.

          And even with some limitations, the Quest is still the most advanced all round affordable fully wireless/untethered all-in-one VR solution on the market.

          You honestly seem to be missing the point here.

  • impurekind

    I always enjoy watching Michael Abrash talk about VR.

  • jimbo5Z

    Abrash, as always, impressive and reassuring.

  • Lucidfeuer

    Can’t wait to see the Oculus Quest’s numbers…though the kool-aid will never run out, this cycle of VR product and interest might. Well at least we’re still getting lots of branding VR projects…

  • oompah

    Waveguide goggles is what will sustain momentum of
    virtual reality , or else it may perish being too uncomfortable.
    Waveguides the magical futuristic tech. that takes away that
    box off ur face & will give u natural feeling of being in that computed world.

    But I wonder , imagine it becomes so refined that govts. start to
    embed it into ur retina at birth & show u what the govt wants u to see
    or believe

  • Kenji Fujimori

    Oculus is getting crappier.. John Carmack on his tweets even hinted at soon leaving, so that is a sign of whats going on internal

    • Mrfox Babbit

      OO! oooo! Maybe he’ll jump over to Valve to begin on the followup to the index.

      • Kenji Fujimori

        Actually I jumped into your mother’s hole

        • Mrfox Babbit

          Look at you go! You typed that out all by yourself! Good for you.
          The old “Mom” hacks, the mark of a slim-witted child. Keep trying though champ.

          • Kenji Fujimori

            Yep keyboard warrior Anglo prick, if we met face to face, you would face a different fate.

          • Mrfox Babbit

            Ah yes, your irrefutable manly boast shows the mark of a true intellectual genius, with a mythical c*ck so massive the swagger is off the charts.

            Boy oh boy you sure are tough and intimidating there champ, nobody F’s with you… pfft what an abject loser poseur.

            Truth of the matter is that undoubtedly you still demand your grilled cheese is browned just right and cut into triangles with crusts removed otherwise a meltdown tantrum ensues.

          • Kenji Fujimori

            Yep, your mother liked it that much, she came back for round three of a creampie

          • Mrfox Babbit

            Uninspired boring rhetoric of a 12 year old.

          • Kenji Fujimori

            Anglo bitch

          • Mrfox Babbit

            You might be somebody one day.

          • Kenji Fujimori

            You too, once you get off unemployment

      • BananaBreadBoy

        Why would he leave for the Index? Carmack’s all about mobile VR, to the point that his joining condition for Oculus was that he be allowed to work on mobile headsets. He’s right where he wants to be at Oculus.

  • Kenji Fujimori

    Same bullshit was also dictated by Magic Leap

  • Jim from Texas

    Why create flat display panels? If Facebook is going to have to develop the technology for next generation display, create a display that is curved in just the right way to fix the issues with optics/lenses.

  • Nerd Man

    Does this mean that we aren’t going to see a new RIFT for 4 to 5 years? (The most important thing that he said is that VR manufacturers are going to have to create the next generation of displays themselves instead of using technology from smartphones. I am not sure that the business case exist for this kind of investment)

  • Darren

    Fixing the multi-focus problem via software to blur the part of the image that weren’t in focus was really interesting. The “Half-Dome” prototype fixed this problem via mechanically moving the display closer or further away. Assuming that “Half-Dome” was a prototype for Rift 2, this seems like a dramatic change in the hardware for the Rift 2.

  • Kenji Fujimori

    Looks Crapulus

  • Mrfox Babbit

    NEAT!

  • dz11

    And then they release the Rift S. Overwhelming.