Quest Pro’s face-tracking capabilities will be quickly put to use to make Meta’s avatars more expressive, but next-gen avatars stand to benefit much more from the new tech.

One of Quest Pro’s big new features is a face-tracking system that uses internal cameras to sense the movement of your eyes and some parts of your face. Combined with a calibration-free machine learning model, the headset takes what it sees and turns it into inputs that can drive the animation of any avatar.

Update (November 18th, 2022): Meta has released the Aura face-tracking sample for developers to experiment with, along with three other developer samples relating to the ‘Movement SDK’ which gives developers a foundation for creating expressive avatars with eye, face, and body-tracking.

In the near-term, this will be put to use with Meta’s existing avatars. And while it certainly makes them more expressive, they still look somewhat goofy.

This is likely the result of the current Meta avatar system not being built with this level of face-tracking in mind. The ‘rigging’—the underlying animation framework of the model—seems not quite fit for the task. Grafting Quest Pro’s face-tracking inputs onto the current system isn’t really doing justice to what it’s actually capable of.

Key Quest Pro Coverage:

Quest Pro Revealed – Full Specs, Price, & Release Date

Quest Pro Review – Impressive Hardware With a Value Proposition That’s Kind of a Mess

Quest Pro Technical Analysis – What’s Promising & What’s Not

Touch Pro Controllers Revealed – Also Compatible with Quest 2

Luckily Meta has built a tech demo which shows what’s possible when an avatar is designed with Quest Pro’s face-tracking in mind (and when almost all of the headset’s processing power is dedicated to rendering it).

Yes, it’s still a bit shaky, but every movement you’re seeing here is being driven by the user making the same motions, including things like puffing out the cheeks or moving the mouth from one side to the other. On the whole it’s a much more complete representation of a face that I’d argue manages to avoid entering into the uncanny valley.

I got to try this demo for myself in my recent hands-on with Quest Pro where I looked into the mirror and appeared as this character (which Meta calls Aura). I came away really impressed that, even with no special calibration, the face I saw in the mirror seemed to mimic whatever motions I could think to make with my face.

I was especially drawn to the detail in the skin. If I squinted and scrunched up my nose I could see the skin around it bunch up realistically, and the same thing when I raised my brow. These subtle details, like the crease in the cheeks moving with the mouth, really add a lot to the impression that this is not just an object in front of me, but something that’s got a living being behind it.

Whether or not the expressions actually look like me when I’m the one behind the mask is another question. Since this avatar’s face doesn’t match my own, it’s actually tough to say. But that the movements are at least plausibly realistic is a first important step toward virtual avatars that feel natural and believable.

Top VR Melee Fighting Game 'Blade & Sorcery' in "final stretch" to Huge 1.0 Update

Meta says it will release the Aura demo as an open source project so developers can see how they’ve attached the face-tracking inputs to the avatar. The company also says developers will be able to use a single toolset for driving humanoid avatars or non-human avatars like animals or monsters without needing to tweak every avatar individually.

Meta says developers will be able to tap a face-tracking API that uses values corresponding to FACS, a well recognized system for describing the movement of different muscles in the human face.

This is an effective system not only for representing faces, but it also forms a useful privacy barrier for users. According to Meta, developers can’t actually get access to the raw images of the user’s face. Insead they get a “series of zero-to-one values that correspond with a set of generic facial movements, like when you scrunch your nose or furrow your eyebrows,” Meta says. “These signals make it easy for a developer to preserve the semantic meaning of the player’s original movement when mapping signals from the Face Tracking API to their own character rig, whether their character is humanoid or even something more fantastical.”

Meta claims even the company itself can’t see the images captured by the headset’s cameras, either internal or external. They are processed on the headset and then immediately deleted, according to the company, without ever being sent to the cloud or to developers.

Newsletter graphic

This article may contain affiliate links. If you click an affiliate link and buy a product we may receive a small commission which helps support the publication. More information.

Ben is the world's most senior professional analyst solely dedicated to the XR industry, having founded Road to VR in 2011—a year before the Oculus Kickstarter sparked a resurgence that led to the modern XR landscape. He has authored more than 3,000 articles chronicling the evolution of the XR industry over more than a decade. With that unique perspective, Ben has been consistently recognized as one of the most influential voices in XR, giving keynotes and joining panel and podcast discussions at key industry events. He is a self-described "journalist and analyst, not evangelist."
  • ViRGiN

    Competition is always better.

    Oh wait, there is no competition. There is clutter disguised under illusion of choice. What’s really out there, is a single leader.

    • Max-Dmg

      Need someone else to make a decent headset that doesnt rip people off in the FB/Apple style.

    • Andrew Jakobs

      Let’s see what Pico gets up to with their Pico 4 Pro.

  • xyzs

    They justify the steep price with the PRO focused claim, then they showcase the headset with a green girl face morphed with a flower…
    I don’t know what a typical professional do with his/her work time from meta’s people point of view but definitely not that… This is clearly a vrchat way of using the “pro” headset though…

    • ViRGiN

      Who are you talking to, loser ?
      I don’t give a f about comments from you.

    • Rupert Jung

      I’m also wandering about that. All that feels a little like a solution in search of a problem. I hope for meta, that there is a market for pro users. With 1-2 hours battery life time. And built-in batteries…

      • Max-Dmg

        They will probably charge extra for the extra battery again lol.

    • Lulu Vi Britannia

      There’s this thing called “entertainment industry”, and believe or not, they’re professionnals too.

      It’s important for motion capture, so it’s good for videogames and movies. If you make an animated movie today, you want something that portrays your actors’ facial motions. And in games and movies, you can most definitely have a green flower girl without looking unprofessional…

      This headset can also be game-changing for V-tubers, as it would give much more life to the characters they’re using.

    • VR5

      Actually seeing the facial expressions with the new (still quite simple) avatars in the keynote was very impressive already. And that’s how they presented it to the audience at large. The point is to enable natural human communication, enhancing presence.

      • Max-Dmg

        Wil this work with those funny looking girls who have lip filler rubber lips?

    • Max-Dmg

      I know a girl who has a green face morphed with a flower.

    • It’s Flower Girl is the only interest avatar I’ve seen so far.

  • Lulu Vi Britannia

    More detailed face animations will always be good, but what’s wrong with the “uncanny valley”? Everyone keeps talking about having ultra-realistic avatars, but there’s really nothing wrong with “goofy”, cartoon-like facial animations. As long as it conveys the user’s real emotions, it’s still a plus.

    • Ben Lang

      Nothing is wrong with cartoon/stylized avatars—they steers clear of the uncanny valley. But ultimately I think there’s a strong desire to see photorealistic representations of the people you care about. The uncanny valley is a real obstacle as you try to move toward photorealism:

      • “Care about”? Sounds like a personal opinion more then anything having to do with reality. I can think of about 50 Disney movies and a dozen Pixar ones that would be happy to prove you wrong.

    • I’d personally prefer something animated and unrealistic. Many of these chat games have been pushing “realistic” avies, but all I see are sheep, afraid to express themselves, so instead they copy the Kardashians. All I got to say is, “Where are all my Monsters at?”, and suddenly people start popping on the creative outfits they REALLY wish they were wearing but were too afraid to show off. Suddenly I feel like I’m surrounded by actual human beings, and not copy/paste robots pretending to be models.

  • This is very cool

  • silvaring

    Anyone else getting reminded of Apple’s face tracking on their new pro line tablets and smartphones? I wonder how Apple will be able to package that tech in a comfortable, sleek VR headset, it will be like this generations i-Mac for productivity and tech workers if they do. Is that the reason why Microsoft isn’t launching new VR devices yet, because they are waiting for Apple to make the first move in the VR productivity space?

    • Why would anyone care? Seriously, Apple is late to every show with overpriced, underwhelming junk. Stop being a brainwashed cult member!

      • BananaBreadBoy

        No one ever cares who does it first, only who does it right. Besides, nothing about Apple lately implies their tech’s gonna be underwhelming. Hell, a regular ass M1 chip would blow any of the snapdragon headsets out of the water.

  • Derek Kent

    Awful and offputting.

  • Lehrit

    This is great looking tech. A lot of deserved animosity towards Meta/Facebook/Zuck, but lets not discount the team of engineers who are leading in this field and are making this happen.

    I don’t like that Meta is laying the groundwork for the future of VR and AR, but I won’t be angry over it. I think it’s kinda funny how social media has been going wild over the whole “meta added legs” meme but conveniently left out the actual innovations like this

  • Sparkette

    I have a Quest Pro; where can I download this tech demo?

  • Arusalan

    You should reread the text.

  • Runesr2

    How cares about this? Where’s the PCVR Lonn review?

    • Jonathan Winters III

      This! Amazing this site has not even acknowledged the game’s release.

  • Jeff is beck

    Get a life nerds

    • kool

      Whats your life like then.

  • Tommy

    That’s pretty cool. I hope the price of the tech comes down for everyone to start including it.