Facebook Shows Off Research Aiming to Deliver Truly Realistic Avatars

The tech is still "years away" from reaching consumers, but the team is working through possible solutions now.

24

Facebook Reality Labs (FRL) chief scientist Michael Abrash believes AR and VR will be the primary way people work, play, and connect in the future. Abrash regularly speaks about when he expects these fundamental milestones in technology to occur, but in a new year-long blog series he wants to drill down to exactly how it’ll happen.

Abrash today revealed in a blog post more about the company’s research surrounding lifelike avatars, something undertaken by the Pittsburgh branch of the company’s skunkworks, Facebook Reality Labs.

Dubbed ‘Codec Avatars’, the Pittsburgh office is using what they call “groundbreaking 3D capture technology and AI systems” to generate lifelike virtual avatars that could provide the basis of a quick and easy personal avatar creator of the future.

The idea, FRL Pittsburgh’s director of research Yaser Sheikh says, is to close physical distances and make creating social connections in virtual reality as “natural and common as those in the real world.”

Image courtesy Facebook

“It’s not just about cutting-edge graphics or advanced motion tracking,” Sheikh says. “It’s about making it as natural and effortless to interact with people in virtual reality as it is with someone right in front of you. The challenge lies in creating authentic interactions in artificial environments.”

It boils down to achieving what the team calls ‘social presence’, and vaulting the uncanny valley to deliver acceptably realistic avatars is something they’ve been working on for years; the team calls the process “passing the ego test and the mother test.”

“You have to love your avatar and your mother has to love your avatar before the two of you feel comfortable interacting like you would in real life. That’s a really high bar,” Sheikh maintains.

A demonstration showing two VR users talking with lifelike avatars gives an interesting look at what the future of VR avatars could be.

 

The company says at this point these sorts of real-time, photorealistic avatars require quite the gear to achieve. The lab’s two capture studios—one for the face, and one for the body—are admittedly both “large and impractical” at this point.

The ultimate goal however is to achieve all of this through lightweight headsets, although FRL Pittsburgh currently uses its own prototype Head Mounted Capture systems (HMCs) equipped with cameras, accelerometers, gyroscopes, magnetometers, infrared lighting, and microphones to capture the full range of human expression.

Image courtesy Facebook

“Codec Avatars need to capture your three-dimensional profile, including all the subtleties of how you move and the unique qualities that make you instantly recognizable to friends and family,” the company says. “And, for billions of people to use Codec Avatars every day, making them has to be easy and without fuss.”

Using a small group of participants, the lab captures 1GB of data per second in effort to create a database of physical traits. In the future, the hope is consumers will be able to create their own avatars without a capture studio and without much data either.

 

At the moment volumetric captures last around 15 minutes, and require a large number of cameras to create the most photorealistic avatars possible. The lab then plans to use these captures to train AI systems so consumers could then quickly and easily build a Codec Avatar from just a few snaps or videos.

Humans come in plenty of different shapes and sizes though, which will be its own challenge to surmount, FRL research scientist Shoou-I Yu says.

“This has taught me to appreciate how unique everyone is. We’ve captured people with exaggerated hairstyles and someone wearing an electroencephalography cap. We’ve scanned people with earrings, lobe rings, nose rings, and so much more,” says Yu. “We have to capture all of these subtle cues to get it all to work properly. It’s both challenging and empowering because we’re working to let you be you,” Yu continues.

SEE ALSO
Leaked Zuckerberg Email Reveals Facebook's XR Strategy, 'Our goal is not only to win, but to accelerate its arrival'

There are still plenty of challenges to address on the way there, Sheikh maintains. One big problem looming on the horizon is ‘deepfakes’, or the act of recreating a person’s appearance or voice to deceive others.

“Deepfakes are an existential threat to our telepresence project because trust is so intrinsically related to communication,” says Sheikh. “If you hear your mother’s voice on a call, you don’t have an iota of doubt that what she said is what you heard. You have this trust despite the fact that her voice is sensed by a noisy microphone, compressed, transmitted over many miles, reconstructed on the far side, and played by an imperfect speaker.”

Sheikh maintains we’re still years away from seeing this level of avatar photorealism, although the lab is currently exploring the idea of securing future avatars through an authentic account, as well as several security and identity verification options for future devices.


Abrash says we’ll be getting more blog posts surrounding optics and displays, computer vision, audio, graphics, haptic interaction, brain/computer interface, and eye/hand/face/body tracking.

This article may contain affiliate links. If you click an affiliate link and buy a product we may receive a small commission which helps support the publication. See here for more information.


Well before the first modern XR products hit the market, Scott recognized the potential of the technology and set out to understand and document its growth. He has been professionally reporting on the space for nearly a decade as Editor at Road to VR, authoring more than 3,500 articles on the topic. Scott brings that seasoned insight to his reporting from major industry events across the globe.
  • gothicvillas

    1gb per second and takes 15min… jeez

  • James Markland

    No! I don’t want realistic you idiots! I want an anime avatar!

    • Lacy

      Most recent poll shows that over 75% men and women are engrossed into online world tasks. Online world is simply becoming bigger and much better and providing lots of make money online opportunities. Work from home internet jobs are becoming poplar and changing individual’s daily lives. Exactly why it is extremely popular? This is because it allows you to work from anywhere and anytime. You receive more time to allocate with all your family and can plan out trips for holidays. People are earning great income of $45000 weekly by utilizing the efficient and smart techniques. Carrying out right work in a right direction will always lead us in the direction of success. You can start to earn from the 1st day at the time you see our website. ENTER NOW >>>>> https://fmfgds.com/3nm

  • Xron

    Looks awesome, though I guess it will still take time for them to become reality.
    It will prosper when we can make them while using our average phone camera. With 3d sensors like in S10 5G.

  • Who’s this?

    Realistically ugly. I love how in the name of diversity they’ve pushed it in media that everyone doesn’t look appealing even though that’s not the case.

    Furthermore people like anime because it does not conform to realities limitations and politics. The more they try to push it, the more people will push back.

    • Nejham Mosquera

      You are just thinking for use cases like VRChat or the Oasis metaverse. But this could be great for businesses and teams working remotely.

  • Bob

    Did anyone noticed the HMD’s those people are wearing looks like the new rift S.

    • Nejham Mosquera

      No, they don’t. If you pay attention to all the headsets that the FRL teams use, they always have extra markers, though it maybe for better tracking.

  • Michael Slesinski

    “groundbreaking”? you mean the same technology vr girlz has used all along?

    • Tesla

      It is a technology from 90s, matrix time. Nothing new. They put tons of cams and film face, then put that on a 3d model and dedicate sets of frames to different gestures. Nothing revolutionary, nothing new to patent (I hope!)

  • sfmike

    Great if you are model pretty. I personally don’t want to look like real world me as I get to be this tired old version of me 24/7 and need a more glamourous virtual break from reality.

  • JesuSaveSouls

    Or we can close our facebook accounts and actually talk and walk with each other.A new but ancient and reinvented technology lost for millenniums.The at of personally communicating.Not ignoring calls to just text.Not replying to a long persona letter with a emoji or thumbs up.But to see a person in high resolution 6dof squared in living color.Like Jesus said to do unto others as you would to you sums up all the law and prophets.

    • dk

      great idea ….close your disqus account

      • crim3

        That would be such a relieve :D

  • Tesla

    It is the question if people want real faces.

  • NooYawker

    Isn’t the whole point of these creepy social settings is to be someone else? Everyone I’ve met online is a model apparently.

  • Dylan

    this is pretty silly actually claiming that you “need” the timed 360 rig. You can get an almost perfect result with a single camera and photogrammetry if you’re fast and your subject can stay expressionless. Everything else can be cleaned up in post and the results would be nearly impossible to discern side by side, we’ve had this a pretty long time, and it’s only getting better with convolutional mesh deform and depth sensors. Still looks great, but realistic generally causes our monkey brains to be averse. There’s a reason semi-cartoon avatars works so well, or why Alita in that new movie looks so appealing. The minor changes from “too perfect” make it “perfect” for our heads to handle.

    • Lucidfeuer

      No you can’t. Photogrammetric tool do not have the capacity of capture faces as accurately, let alone in motion. In fact that’s what was used in L.A. Noir too, hence the cost and “failure” of the project for Rockstar (though an amazing game).

      • Dylan

        yeah, okay then. I’ve been doing this for literal years – photogrammetry has matured.

        • Lucidfeuer

          Nope you’ve never done it, your first text is pretty explicit about it, you don’t know shit about what you’re talking about…if we’re talking about the same thing which is model animation and tracking real-time or not…

    • Justin Sena

      Alita in that movie looked appealing to you?? She looked absolutely horrid to me.

  • Farguns

    Yeah miss me with hyper realism. I don’t want to see some black person’s highly detailed porous face in VR. Why escape reality but bring along all the imperfections.

  • Lucidfeuer

    The fact that they have to use a now 15 years old face capture technic to even make the avatar completely defeats the purpose. But at least the demo is reinsuring about the possibility.

  • I’ve read on CBS that the rig is currently made by 180 cameras, that streams an incredible amount of TBs for seconds. We’re still years away from an usable solution, but this is really intriguing nonetheless