During the Connect 2021 conference last week, Meta Reality Labs’ Chief Scientist, Michael Abrash, offered a high-level overview of some of the R&D that’s behind the company’s multi-billion dollar push into XR and the metaverse.

Michael Abrash leads the team at Meta Reality Labs Research which has been tasked with researching technologies that the company believes could be foundational to XR and the metaverse decades in the future. At Connect 2021, Abrash shared some of the group’s very latest work.

Full-body Codec Avatars

Meta’s Codec Avatar project aims to achieve a system capable of capturing and representing photorealistic avatars for use in XR. A major challenge beyond simply ‘scanning’ a person’s body is getting it to then move in realistic ways—not to mention making the whole system capable of running in real-time so that the avatar can be used in an interactive context.

The company has shown off its Codec Avatar work on various occasions, each time showing improvements. Initially the project started off simply with high quality heads, but it has since evolved to full-body avatars.

The video above is a demo representing the group’s latest work on full-body Codec Avatars, which researcher Yaser Sheikh explains now supports more complex eye movement, facial expressions, and hand and body gestures which involve self-contact. It isn’t stated outright, but the video also shows a viewer watching the presentation in virtual reality, implying that this is all happening in real-time.

With the possibility of such realistic avatars in the future, Abrash acknowledged that it’s important to think about security of one’s identity. To that end he says the company is “thinking about how we can secure your avatar, whether by tying it to an authenticated account, or by verifying identity in some other way.”

Photorealistic Hair and Skin Rendering

While Meta’s Codec Avatars are already looking pretty darn convincing, the research group believes the ultimate destination for the technology is to achieve photorealism.

Above Abrash showed off what he says is the research group’s latest work in photorealistic hair and skin rendering, and lighting thereof. It wasn’t claimed that this was happening in real-time (and we doubt it is), but it’s a look at the bar the team is aiming for down the road with the Codec Avatar tech.

Clothing Simulation

Along with a high quality representation of your body, Meta expects clothing with continue to be an important way that people want to express themselves in the metaverse. To that end, they think that making clothes act realistically will be an important part of that experience. Above the company shows off its work in clothing simulation and hands-on interaction.

High-fidelity Real-time Virtual Spaces

While XR can easily whisk us away to other realities, teleporting friends virtually to your actual living space would be great too. Taken to the extreme, that means having a full-blown recreation of your actual home and everything in it, which can run in real-time.

Well… Meta did just that. They built a mock apartment complete with a perfect replica of all the objects in it. Doing so makes it possible for a user to move around the real space and interact with it like normal while keeping the virtual version in sync.

So if you happen to have virtual guests over, they could actually see you moving around your real world space and interacting with anything inside of it in an incredibly natural way. Similarly, when using AR glasses, having a map of the space with this level of fidelity could make AR experiences and interactions much more compelling.

Apple: Vision Pro Will Have 600+ Native Apps Ready at Launch

Presently this seems to serve the purpose of building out a ‘best case’ scenario of a mapped real-world environment for the company to experiment with. If Meta finds that having this kind of perfectly synchronized real and virtual space becomes important to valuable use-cases with the technology, it may then explore ways to make it easy for users to capture their own spaces with similar precision.

Continued on Page 2 »


This article may contain affiliate links. If you click an affiliate link and buy a product we may receive a small commission which helps support the publication. See here for more information.

Ben is the world's most senior professional analyst solely dedicated to the XR industry, having founded Road to VR in 2011—a year before the Oculus Kickstarter sparked a resurgence that led to the modern XR landscape. He has authored more than 3,000 articles chronicling the evolution of the XR industry over more than a decade. With that unique perspective, Ben has been consistently recognized as one of the most influential voices in XR, giving keynotes and joining panel and podcast discussions at key industry events. He is a self-described "journalist and analyst, not evangelist."
  • Adam

    Just imagine someone stealing your girlfriends photorealistic avatar… Enjoy all of your coworkers having a swing with her in the “MetaBrothel Extreme Penetration — Cuckold Edition”.

    • ViRGiN

      This spect has been covered by Mark.

    • Plou

      It can already happen. It’s called imagination.

      • silvaring

        email me your imagination archive when you get a chance, I want to download it into my brain.

    • kontis

      You are not describing anything new. It’s not even close to being as big of a deal as you think it is.

      We had photoshop dramas like this in the 90s with just pictures, some wanted to delegalize photoshop and after a few years no one cared anymore.

      Then people started ripping 3D models of real actors from video games 20 years ago and doing nsfw machinima with them. With current high fidelity models it’s even more widespread. Sometimes there is a bit of drama but nothing substantial (and sometimes it’s on official level, like the Ellen Page’s model in Beyond two souls being naked in the developer mode).

      Then we had Deep fakes drama. As before no one cares anymore.

      That “photorealistic avatar” will be nothing but 3D model and never fully accurate and never behaving like real person, so after a few clickbaity articles no one will care, because it will feel like a more elaborate cringy photoshop and just like with the uncanny valley, when people feel something is off they lose interest.

      • Andrew Jakobs

        Uhm actually realistic moving digital characters are already possible at this point, and the last pieces of removing unnatural motions is already being worked on and perfected. In only 2 years or so you really wouldn’t be able to tell the difference, if people want to.

    • Andrew Jakobs

      Well, people downvote you, but it certainly is something that has to be taken into account. Deepfakes already are a problem, but with ultrarealistic avatars it can even go another step further.

    • sfmike

      The real question is will FB/Meta ever allow a MetaBrothel to ever exist or will any expression of adult sexuality get you kicked off the platform like we have seen enforced by FB over the years.

      • ViRGiN

        Valve doesn’t allow any erotica, unless it’s all furry/weeb or nazi fetish. Why would Meta be any different?

    • Anonymous

      I think this is simply destined to happen no matter what as technology improves. It will just be a way of life and not necessarily a bad thing, or maybe even a good thing.

      I will just admit, I look forward to the day it is simple to make deepfakes or fool around in adult ways with the avatar of whatever person I have a fancy for. I’m pretty sure you would try it too if given the choice and it were effortless. As would 99% of other men and some women, though few would admit it.

      Human nature will make it happen.

      • guest

        Covet thou neighbors wife is no longer a comandment when you can also buy a simulation of the inside of their house from Facebook AKA Meta…

  • VRFriend

    Game engines have this tech since 10 years and it did not cost billions of dollars.
    CryEngine, Unreal Engine. C++ code publicly available :) Meta added some more stuff. All these companies copy each other.
    The wall to capture the avatar is such an obvious thing. Costs maybe 30 000 USD to fill with remotely controlled cameras. Nothing new. I worked on this in the year 2000, before Facebook even existed. It is interesting Meta is pushing this, but I do not see anything new here beside connecting this with VR social platform.

    • ViRGiN

      So what are you saying exactly?

      Did you tell them they overpaid about 33333%?

    • kontis

      The things game engines already do was NOT the point of this article or this research and it is NOT where the money went.

      This is like telling an artists that he shouldn’t be proud of his art because paint was invented centuries ago.

    • Andrew Jakobs

      Uhm, bullshit, as an Unreal Engine lover I can honestly say they didn’t have these realistic human avatar tech until they announced Meta humans, which they only added this year, and it certainly wasn’t realtime..

      • VRFriend


    • benz145

      Before shititng on people’s work, perhaps you should know more about it?

      Their innovation in codec avatars is not about creating a photogrammetry model. It’s animating the model well enough to cross the uncanny valley, in-real time, through inputs like face, head, and hand tracking, which is exceptionally difficult. You can learn more about the way the system works here: https://www.roadtovr.com/facebook-expands-on-hyper-realistic-virtual-avatar-research/

  • 144Hz

    This is the beginning of the end.

  • Lucidfeuer

    The only interesting (but very limited) thing I see is the contextual AI, which other just call basic computer vision interfacing. The rest is making them look like Magic Leap, they’re not developing anything substantial, I mean what’s that thing about cloth simulation?

    • Lester

      You may not see it but I certainly do. VR has always been a systems integration and software problem. Otherwise it’s just It’s screens strapped to your eyes. They are solving those problems. I think it will be a huge deal in the long term. But I can’t see it’ll be good for any of us. It is the beginning of a boring, dystopian distraction that our children will be drawn into on a mass scale, not like 1984, but like brave new world. Fake dreams, fake ads, fake jobs, fake life to hide the lowering standards in this one.

  • Arturs Gerskovics

    I personally think they went overboard. Too much hype about things we maybe will have in 10-15 years. They could have spent a little segment on that and the rest about what we can have now or at least VERY soon. At the end of the day, Quest is a mobile VR/AR and it just can’t do the same things what PC can do. So, any cloth simulations or vast high fidelity open worlds will require a decent power which just cant be mounted on your head. It needs an outside source to feed. Maybe META MACHINE could help this. Zuck could make $200-300 META Machines matching RTX3090 and i9 with crazy cooling to drive his fantasy METAVERSE.

    • kontis

      Connect keynotes are no longer for VR nerds and geeks. They are now for investors and general public. They are trying to move to the mainstream with stupid ads. We are far too niche for Facebook.

      It’s a known tactic and it often works. Every time Apple shows new technically absurd (and scientifically incorrect) graph comparing their product to some undefiled competing solution the tech people laugh and cringe how stupid and misleading it is, but general public is in awe and gets hyped like crazy. Blame people for being stupid and eating these things up. Facebook is just falling at execution of the same BS method.

      • psuedonymous@mailinator.com

        The OC/FC events are for VR geeks and nerds who are interested in VR research, not consumer hardware. The consumer hardware fans will get dedicated launch hyper events when those devices are unveiled just like any other electronic device, whilst the VR nerds get Connect showing an overview of research including and beyond the papers already published by FRL.

  • david vincent

    Impressive but once again : who wants to have his IRL face & body as his avatar ?

    • ViRGiN

      Everyone but furries and weebs. Good luck working with real people using your favorite Genshin Impact avatar.

      Who wants to be a furry, hanging out with weebs?

      • John Doe

        You forgot incel gamers xd

        Also, remote working will never be a thing, because companies hate it ;p

    • I’d have to agree, most of us aren’t models. But maybe switching bodies and still retaining realism is possible?

    • Kenny Thompson

      People in professional settings.

  • Mrfox Babbit

    Neckbeards and incels rejoice! The true question lies in can they render a realistic fedora on a jaunty angle and subservient girlfriend AI?

    • John Doe

      Huh, I though video gamers are neckbeards and incels xd

  • Kenny Thompson

    Wow, I didn’t realize that the white room was actually in VR as well until the 3rd watch. Crazy.