Researchers at Facebook Reality Labs today published new work showcasing a prototype headset which has external displays to depict the user’s eyes to others outside of the headset. The goal is to allow eye-contact between the headset wearer and others in an effort to make it less awkward while wearing a headset and communicating with someone in the same room.

One of my favorite things to do when demoing an Oculus Quest to someone for the first time is to put on the headset, activate its ‘passthrough view’ (which lets me see the world outside of the headset), and then walk up and shake their hand to clearly reveal that I can see them. Because Quest’s cameras are at the four corners of the visor, it’s not easy to imagine that there would be any way for the user to see ‘through’ the headset, so the result from the outside seems a bit magical. Aftward I put the headset on the person and let them see what I could see from inside!

But this fun little demo reveals a problem too. Even though it’s easy for the person in the headset to see people outside of the headset, it isn’t clear to people outside of the headset when the person in the headset is actually looking at them (rather than looking at an entirely different virtual world.

New Live Captions Feature on Vision Pro Could Lead the Way to Real-time Translation

Eye-contact is clearly a huge factor in face-to-face communication; it helps us gauge if someone is paying attention to the conversation, how they’re feeling about it, and even if they have something to say, want to change the topic, or leave the conversation entirely. Trying to talk to someone whose eyes you can’t see is uncomfortable and awkward, specifically because it robs us of our ingrained ability to detect this kind of intent.

But as VR headsets become thinner and more comfortable—and it becomes easier to use passthrough to have a conversation with someone nearby than taking the headset off entirely—this will become a growing issue.

Researchers at Facebook Reality Labs have come up with a high-tech fix to the problem. Making use of light-field displays mounted on the outside of a VR headset, the so called ‘reverse passthrough’ prototype system aims to show a representation of the user’s eyes that’s both depth and direction accurate.

Image courtesy Facebook Reality Lab

In a paper published this week for SIGGRAPH 2021, Facebook Reality Labs researchers Nathan Matsuda, Joel Hegland, and Douglas Lanman, detailed the system. While to external observers it appears that the headset is very thick but transparent enough to see their eyes, the apparent depth is an illusion created by a light-field display on the outside of the headset.

If it was instead a typical display, the user’s eyes would appear to float far away from their face, making for perhaps a more uncomfortable image than not being able to see them at all! Below researcher Nathan Matsuda shows the system without any eyes (left), with eyes but no depth (middle), and with eyes and depth (right).

The light-field display (in this case a display which uses a microlens array), allows multiple observers to see the correct depth cues no matter which angle they’re standing at.

What the observers see isn’t a real image of the user’s eyes however. Instead, eye-tracking data is applied to a 3D model of the user’s face, which means this technique would be limited by how realistic the model is and how easy it is to acquire for each individual.

Of course, Facebook has been doing some really impressive work on that front too with their Codec Avatars project. The researchers mocked up an example of a Codec Avatar being used for the reverse passthrough function (above), which looks even better, but resolution is clearly still a limiting factor—something the researchers believe will be overcome in due time.

Facebook Reality Labs Chief Scientist, Michael Abrash admits he didn’t think there was much merit to the idea of reverse passthrough until the researchers further proved out the concept.

“My first reaction was that it was kind of a goofy idea, a novelty at best,” Abrash said in a post about the work. “But I don’t tell researchers what to do, because you don’t get innovation without freedom to try new things, and that’s a good thing, because now it’s clearly a unique idea with genuine promise.”

– – — – –

It might seem like a whole lot of work and extra hardware to solve a problem that isn’t really a problem if you just decided to use an AR headset in the first place. After all, most AR headsets are built with transparent optics from the outset, and being able to see the eyes of the user is a major benefit when it comes to interfacing with other people while wearing the device.

But even then, AR headsets can suffer from ‘eye-glow’ which obstructs the view of the eye from the outside, sometimes severely, depending upon the optics and the angle of the viewer.

Image courtesy DigiLens

AR headsets also have other limitations that aren’t an issue on VR headsets, like a limited field-of-view and a lack of complete opacity control. Depending upon the use-case, a thin and light future VR headset with a very convincing reverse passthrough system could be preferable to an AR headset with transparent optics.

Newsletter graphic

This article may contain affiliate links. If you click an affiliate link and buy a product we may receive a small commission which helps support the publication. More information.

Ben is the world's most senior professional analyst solely dedicated to the XR industry, having founded Road to VR in 2011—a year before the Oculus Kickstarter sparked a resurgence that led to the modern XR landscape. He has authored more than 3,000 articles chronicling the evolution of the XR industry over more than a decade. With that unique perspective, Ben has been consistently recognized as one of the most influential voices in XR, giving keynotes and joining panel and podcast discussions at key industry events. He is a self-described "journalist and analyst, not evangelist."
  • VRFriend

    Nonsense. worthless.

    • Barbar Jellison

      ‘m able to make 80 dollars per hour to complete few tasks on the computer…~no171~I not ever imagined that it’s even achievable however one of my bosom friend was receiving $27k only in a month easily doing this super offer and also she had convinced me to avail…~no171~Find further details on this web >>>

    • Cless

      I mean… I guess it has a use as investigation…? But I have to agree, no use in the current market whatsoever. To be fair, I don’t think any nonVR AR is or will be relevant at all until the 2030s… at a MINIMUM.

  • Carlos

    User: Please, try to improve FOV, bring back the old IPD settings, make batteries last longer, make it lighter…

    • kontis

      Facebook – we can’t make AR glasses with current technology, so let’s fix this “important” issue that millions of people experience for 100 years when wearing sunglasses and never care (or even consider it a feature), because we are a stupid megacoporation that overinvested and doesn’t know what to research.

      • Linda Thompson

        I’m able to produce 88 US dollars per hour to do easy tasks on the internet…~gy106~I’ve not ever believed that it could be achievable however my confidant mate produced $27,000 only within three weeks easily doing this top opportunity and also she satisfied me to try…~gy106~Read additional instructions by going this link >>>

  • dk they could just glue this to the front of the quest 6

  • JB1968

    Absolutely useless. This is pure definition of “gimmick” in my universe.

    • David

      To be fair, it’s just a research prototype. I doubt it’ll be included in any actual product for a long while, if ever.

      • d0x360

        Yes it’s a research prototype… And who do you think pays for that? Facebook?

        Nope, we do when we buy an Oculus product. All that wasted r&d money MUST be recouped… So if we weren’t paying for useless crap like this we could either get an HMD that uses better hardware OR one that uses the hardware it uses now but it would be $100 cheaper.

        Sound fair?

        • David

          Quest 2 is already super cheap compared to other headsets on the market, and it wouldn’t surprise me if the cost was already being subsidized to keep it cheap for consumers. I seriously doubt that R&D experiments like this are going to materially affect the price of any future headsets. Facebook already dumps a ton of money into stuff like this simply because they can, because they’re swimming in cash. I really wouldn’t worry about it.

  • Kevin White

      Bravo! Thanks for a good belly laugh :)

    • It’s much cheaper and effective the same!

      • Natasha Ponce

        I am able to create 80 dollars/hr to complete easy tasks staying at home…~og154~I not ever thought like it can be even achievable but my greatest mate was making $25,000 within five weeks simply working this leading opportunity & she had satisfied me to avail…~og154~Visit for fresh info visiting this website >>>

  • David

    One of my favorite things to do when demoing an Oculus Quest to someone for the first time is to put on the headset, activate it’s ‘passthrough view’

    Minor thing, but the word “its” here shouldn’t have an apostrophe. It should say “activate its ‘passthrough view'”. The version with the apostrophe always means “it is” or “it has”.

    • benz145

      Thanks, my mistake!

  • d0x360

    What a collosal waste of ****ing money. We pay for this r&d when we buy an HMD and here we have Facebook just wasting probably millions of dollars for a feature absolutely nobody cares about..

    As soon as valve releases an HMD with inside out tracking I’m selling my Rift S. I’ve had enough with Oculus.

  • xyzs

    Hey Facebook, it’s on the OTHER SIDE of the headset that you need to improve the display technology….

  • xyzs

    “But as VR headsets become thinner and more comfortable—and it becomes easier to use passthrough to have a conversation with someone nearby than taking the headset off entirely—this will become a growing issue.”

    Well, nearly a __decade__ after the dk1, the form factor is still __extacly__ the same with ZERO progress done, so maybe let’s start with doing some progress with current products before thinking what the future in 30 years will need ?

    • benz145

      If you look at the original paper, the researchers believe this tech will become mature in line with other improvements in form-factor like folded holographic optics:

      • xyzs

        Yep, I would love to see these holographic optics be a real product.
        But it’s not before at least 10 years if even.
        Already the half dome prototype v3 that we know for many years is not even close to being a consumer product for a while.
        So far the Quest 2 is still a very primitive hardware and that’s all we have.

  • Gildahl

    So last week FB was creating a metaverse and this week it is creating cyborgs. Will these eyes automatically roll whenever someone criticizes FB?

  • sfmike

    Waste of time and money. I agree with others, work on FOV. Personally I would pay $100 more for OLED blacks to come back but they don’t seem to listen or care.

  • I don’t need people thinking I’m staring at them when I’m trying to open a door

  • Foreign Devil

    All these resources being put into something I really don’t care about. . . FB is too social focused. .. I guess I’m the type of VR user that is in the minority but I use it for productivity and solo gaming. I don’t like kid filled social VR games for the most part.

  • Ian Shook

    This is sort of nuts. In a good way. I totally hate FB too, BUT, you don’t have to hate their engineering team. Do you guys realize this is one of the first glimpses of a lightfield display? That’s awesome! Lightfield displays are so cool and they’ve been stuck in tech labs until now. The FB team is doing some amazing R and D, regardless of who they work for. There are very few, if ANY videos out there of a functioning LF display.

  • mirak

    that’s awfull

  • Rupert Jung

    Do they have any advanced research going on in how to integrate a comfortable headstrap? ;) No seriously, would rather love to see their varifocal display in action, along with eye tracking and a higher FOV. Oh yeah, and it would be great if I could set my IPD again, like in the first version…

  • This is an amazing project. For anyone complaining, this is research, and it should go in whatever directions possible if we want innovations to happen. Maybe this will never be used, or maybe part of the technology developed here will be useful for something else. Kudos to Abrash for having made this possible.

  • crim3

    Sun glasses manufacturers are like “what problem?”

  • JustNiz

    I can’t believe people waste time studying this. Given this “feature” inevitably incurs extra cost, weight, size and power consumption, I would actively avoid buying any headset that has this.

  • This is the first time I’ve seen somebody take a human face *INTO* the Uncanny Valley, instead of out of it.

    I think the whole article could be summed up with this slightly out of context quote:
    “a whole lot of work and extra hardware to solve a problem that isn’t really a problem”

    On the bright side, I can see some novelty electronic glasses being made out of this tech in the distance future. Think a high-tech pair of Googly Eye glasses!