Meta unveiled two new VR headset prototypes that showcase more progress in the fight to solve some persistent technical challenges facing VR today. Presenting at SIGGRAPH 2023, Meta is demonstrating a headset with retinal resolution combined with varifocal optics, and another headset with advanced light field passthrough capabilities.

Butterscotch Varifocal Prototype

Revealed in a developer blogpost, Meta showed off a varifocal research prototype that demonstrates a VR display system which provides “visual clarity that can closely match the capabilities of the human eye,” says Meta Optical Scientist Yang Zhao. The so-called ‘Butterscotch Varifocal’ prototype provides retinal resolution of up to 56 pixels per degree (PPD), which is sufficient for 20/20 visual acuity, researchers say.

Since its displays are also varifocal, it can support from 0 to 4 diopter (i.e. infinity to 25 cm), and matching what researchers say are “the dynamics of eye accommodation with at least 10 diopter/s peak velocity and 100 diopter/s2 acceleration.” The pulsing motors below control the displays’ focal distance in an effort to match the human eye.

Varifocal headsets represent a solution to the vergence-accommodation conflict (VAC) which has plagued standard VR headsets, the most advanced consumer headsets included. Varifocal headsets not only include the same standard support for the vergence reflex (when eyes converge on objects to form a stereo image), but also the accommodation reflex (when the lens of the eye changes shape to focus light at different depths). Without support for accommodation, VR displays can cause eye strain, make it difficult to focus on close imagery, and may even limit visual immersion.

Check out the through-the-lens video below to see how Butterscotch’s varifocal bit works:

Using LCD panels readily available on the market, Butterscotch manages its 20/20 retinal display by reducing the field of view (FOV) to 50 degrees, smaller than Quest 2’s ~89 degree FOV.

Although Butterscotch’s varifocal abilities are similar to the company’s prior Half Dome prototypes, the company says Butterscotch is “solely focused on showcasing the experience of retinal resolution in VR—but not necessarily with hardware technologies that are ultimately appropriate for the consumer.”

“In contrast, our work on Half Dome 1 through 3 focused on miniaturizing varifocal in a fully practical manner, albeit with lower-resolution optics and displays more similar to today’s consumer headsets,” explains Display Systems Research Director Douglas Lanman. “Our work on Half Dome prototypes continues, but we’re pausing to exhibit Butterscotch Varifocal to show why we remain so committed to varifocal and delivering better visual acuity and comfort in VR headsets. We want our community to experience varifocal for themselves and join in pushing this technology forward.”

SEE ALSO
Meta Reportedly Set to Unveil Prototype AR Glasses 'Orion' at Connect 2024

 

Flamera Lightfield Passthrough Prototype

Another important side of making XR more immersive is undoubtably the headset’s passthrough capabilities, like you might see on Quest Pro or the upcoming Apple Vision Pro. The decidedly bug-eyed design of Meta’s Flamera research prototype is looking for a better way to create more realistic passthrough by using light fields.

Research Scientist Grace Kuo wearing the Flamera research prototype | Image courtesy Meta

In standard headsets, cameras are typically placed a few inches from where your eyes actually sit, capturing a different view than what you’d see if you weren’t wearing a headset. While there’s a lot of distortion and placement correction going on in standard headsets of today, you’ll probably still notice a ton of visual artifacts as the software tries to correctly resolve and render different depths of field.

“To address this challenge, we brainstormed optical architectures that could directly capture the same rays of light that you’d see with your bare eyes,” says Meta Research Scientist Grace Kuo. “By starting our headset design from scratch instead of modifying an existing design, we ended up with a camera that looks quite unique but can enable better passthrough image quality and lower latency.”

Check out the quick explainer below to see how Flamera’s ingenious capture methods work:

Now, here’s a comparison between an unobstructed view and Flamera’s light field capture, showing off some pretty compelling results:

As research prototypes, there’s no indication when we can expect these technologies to come to consumer headsets. Still, it’s clear that Meta is adamant about showing off just how far ahead it is in tackling some of the persistent issues in headsets today—something you probably won’t see from the patently black box that is Apple.

You can read more about Butterscotch and Flamera in their respective research papers, which are being presented at SIGGRAPH 2023, taking place August 6th – 10th in Los Angeles. Click here for the Butterscotch Varifocal abstract and Flamera full paper.

Newsletter graphic

This article may contain affiliate links. If you click an affiliate link and buy a product we may receive a small commission which helps support the publication. More information.


Well before the first modern XR products hit the market, Scott recognized the potential of the technology and set out to understand and document its growth. He has been professionally reporting on the space for nearly a decade as Editor at Road to VR, authoring more than 3,500 articles on the topic. Scott brings that seasoned insight to his reporting from major industry events across the globe.
  • Lol Flamera looks like some next level sci-fi piece of gear. If I don’t see soldiers running with these visors in 50 years while having real-life wallhacks, my disappointment will be immeasurable and my day will be ruined.

    • CrusaderCaracal

      Yeah looks straight out of cyberpunk 2077, can’t wait

  • jiink

    The benefits of Flamera over Rift S / Quest passthrough are clear, but I’m not understanding what the benefit is over two normal RGB cameras lined up with the user’s eyes like Lynx or Varjo XR3. Is it that this Flamera system positions a ‘virtual’ aperture right where the user’s eyes are located depth-wise (1:34 in video)? Does it require eye tracking and cause it to be as if a normal camera was steering with the eye?

    • Blaexe

      The cameras are never at the same position as your eyes and always need correction, leading to artifacts. Latency is also a factor when it comes to post-processing.

      • David

        They had one prototype that had a mirror front, and cameras mounted in front of the mirror pointing towards the mirror, precisely opposite of where the eyes are, so the reflected cameras were essentially in the same position as the user’s eyes.

        But then you have a blind spot from the camera and arm holding it up.

        • John G

          And if you look down or to the side, then they aren’t aligned.

          The light field allows correction as there is more information collected

          • David

            Your pupil position changes by a few mm as you move your eye around, so the image won’t be precisely the same in your periphery, but by putting the reflected camera in the center of your eyeball, it should be correct in the central part of your vision no matter where you look.

            The headset using the light field camera will have the same issue if it isn’t also using a light field display or eye tracking. My understanding is that it’s just stitching together multiple slightly different camera views into a single 2D image for each eye.

          • John G

            I’d have to see a diagram, I can’t quite figure that out.

            Regarding a regular headset, I think it’s more than that? IPD itself can be several mm. The eyeball is a couple of CM, so if you look to the left I think it’s more, but I’d have to check.

            I used to be frustrated that all of the headsets with external cameras never seem to put the camera where eyes were, but in thinking about it, I think it’s because the scene would only be accurate if you were looking straight ahead.

            I kinda get used to the Quest pass through, but I don’t use it for navigation, just for grabbing the controllers :)

            > My understanding is that it’s just stitching together multiple slightly
            different camera views into a single 2D image for each eye.

            Lightfields are much more than that. You really have full parallax even by moving your head a few feet. I’ve seen it demonstrated at it really felt like “real VR”

            So a few MM or even a CM should be easy to do.

          • psuedonymous

            What’s you’re describing is a (decades) old technique called a CAMERA/AREMAC arrangement. It’s VERY field of view limited due to the size of the mirror required, and setups that do not match the sensor size to the display size also run into issues with distortion due to dissimilar optics.

  • ViRGiN

    Surprised noone has said it yet.
    This is OBVIOUSLY in response to Apple Vision, and growing danger from SteamVR.
    Amazing what little competition can cause huh?

    • Andrew Jakobs

      This is something they have already been working on for years, so it has got nothing to do with the Apple vision Pro. They have been showcasing these type of prototypes for years during siggraph and other graphix/VR related expositions.

      • ViRGiN

        LMAO I know. Just like they have been working on Quest 3 for years. Yet Q3 announcement was “agreed” by the commenters to be made in response to Vision Pro?

  • That’s very cool. Plus I like the design of Flamera

  • Ookami

    Pretty interesting stuff. While I think Varifocal lenses will go a long way towards immersion, I think the true future for vr would be in light-field displays instead of screens and lenses (look up CReal’s work).
    The lightfield passthrough looks really cool, though.

  • Anfronie

    “Retina display” is a dumb apple marketing term for simple minded consumers.

  • Naruto Uzumaki

    they should put a few 100 billion $ in researching how to not put the battery in the front on the quest 3

    • david vincent

      Yeah, filar power transmission technology is hard.
      Also if they could put another few billions in display port researching…

      • ViRGiN

        no

  • Naruto Uzumaki

    1 question who is going to buy these headsets if the quest pro didn t do well

    • Nepenthe

      Nobody since they’re not production headsets.

    • Ookami

      They’re just for prototyping features.

  • ShaneMcGrath

    That’s great and all but I would much prefer they give us more FOV, Even if it’s just another 15 degrees.
    Resolutions are already decent but the FOV still sucks and takes away a lot of immersion.

    • Ookami

      I imagine we’re going to get that in time. There was one company that recently showed almost 100% fov covering displays, using 4 lenses and screens. And it’s very small too.
      Of course it’ll be a while for that tech to trickle into consumer release, but we’ll get there.

  • david vincent

    Oh yes, yet another headset with cutting edge technology wasted by the lack of display port.

    • ViRGiN

      yet another nerd who refuses to comprehend nobody cares except nerds like you who never step out of their basement playing their steamvr indie game collection.

    • Ookami

      They’re just prototypes. Don’t expect it to have all the bells and whistles of a consumer-release headset. On the other hand, consumer headsets might have more dialed back features because of cost. Prototypes are an interesting window into future tech, but are rarely an accurate picture of what you’re actually going to get.

      • david vincent

        You missed my point.

        • Ookami

          You missed my point: just because these prototypes don’t have display port doesn’t mean it won’t be in a release version. Unless you’re talking about the lack of display ports on graphics cards?

    • CrusaderCaracal

      think you missed the part where they said “prototype” it’s obviously not being released yet, or even all for that matter. it’s literally just a demo for what they could implement

      • david vincent

        Another one missing my point, I talk about “future headsets”, not about those particular prototypes.

  • Shuozhe Nan

    Wish Lytro was still around, same problem need to be solved again now :(

  • xyzs

    It feels like watching an oculus connect from 6 years ago….

  • Rupert Jung

    That’s really super-cool, especially varifocal prototypes they have shown in the last years. But what is this all about when they even stripped away essential eye tracking from the Quest 3?

  • Anfronie

    This is awesome tech!!!