At GDC 2013 I saw a cool demo at the AMD booth called FaceX. Created by a company called Mixamo, FaceX tracks a player’s face using a standard webcam and can use that data to animate a 3D model in real-time. While this tech could be used as cheap motion capture to animate computer-controller characters, the much more exciting implication would be real-time facial animation for multiplayer games where teammates communicate with voice chat.

FaceX could be an elegant solution to increasing immersion in current multiplayer games that aren’t yet making the leap to full blown VR.

In theory, the system is perfect for a PC gamer. Most likely they’ve already got a microphone/headset to communicate with teammates. If they’re on a laptop, they already have a camera, or if they’re using a desktop, a cam can be picked up on the cheap — either way, they will already be close to the camera for accurate tracking. Developers wouldn’t even need to sync the audio to the facial animation as they are both being delivered in real time. There’s a huge audience of PC gamers to which this technology could be deployed quickly and inexpensively.

Using this tech with console gamers would be a bit more complicated, but not impossible. Current console cameras like the Kinect and PlayStation Eye probably don’t have the necessary resolution to accurately track gamers’ faces as far as the couch. Higher resolution cameras would be an easy fix, and we’ll probably see that from the Kinect 2 and the PS4 Eye.

FTC Sues to Block Meta's Acquisition of Top VR Fitness App Developer

FaceX tracks more than just mouth and eye movement. It uses the incoming data to convincingly animate cheeks and other subtle face muscles thanks to machine-learning analysis of over 30 test subjects, Mixamo told me.

You can see that it’s a bit jumpy in the video as I turn off to the side to ask the rep questions and look down at my camera, but front-on, where PC gamers are looking all the time, it works quite well, and I would expect it to get better over time.

I can imagine it now: hunkered down in Battlefield 3 with bullets whizzing over my head. The M-COM Station objective is just a few yards away, but the defensive line has me and a teammate suppressed. My teammate, who is communicating with me over voice chat, turns to me and says, “Drop some smoke and head straight for the station, I’ll flank left and take them out,” all the while, I can actually see his mouth moving in real time — and hear the audio emanating from his avatar. This would be significant step forward over simply hearing a disembodied voice coming from nowhere.

Another great place for FaceX would be slower MMORPG games like Second Life or World of Warcraft where players frequently have extended face-to-face interaction through trading, meetings, and general banter. Second Life with FaceX would be fun, but the novelty of seeing a non-human avatar (like a Troll or Orc in World of Warcraft) animated as though you are talking through them would be very cool indeed.

While this tech would be great for virtual reality as well, there are some issues. The near-term future of virtual reality will likely revolve around head mounted displays like the Oculus Rift. This means that the eyes are always going to be obscured to a normal webcam. There was ways around this, like small eye-tracking cameras inside of the HMD, but that adds an extra hardware requirement which quickly begins to shave down the potential audience. There’s also the issue of head-tracking wherein players using head mounted displays will not always be looking directly at a computer monitor in front of them. Thus their faces will sometimes be obscured when they turn away from the camera and tracking would be lost.

Meta to Merge 'Venues' Event Space into 'Horizon Worlds' Social VR Platform

But mainstream virtual reality gaming is still a little ways off. Meanwhile, there are tons of games that could benefit from this tech; Mixamo tells me that FaceX will be launched in the next 6 months.

This article may contain affiliate links. If you click an affiliate link and buy a product we may receive a small commission which helps support the publication. See here for more information.

  • John Dewar

    I actually think an HMD gives you a leg up in collecting this kind of data. Because it’s attached to your face, you don’t have this issue where when you look away from the webcam, the tracking solution falls apart. I was talking to Paul Debevec about this idea recently and he suggested we could use pressure sensors placed around the contact region where the padding on the goggles touches your face to gather expression data. Another idea I had is using an infrared sensor in the goggle to measure blood flow to your face, so we could get blushing and anger in there as well. Eye tracking is a big challenge in the confines of the Oculus design but I’m sure there’s a way to squeeze it in there. And there you also have an advantage of control, where the eyes are fixed relative to the camera and you know what the light levels inside the mask are and possibly can put in an infrared light to blast your irises and improve tracking fidelity. If you hunt around online you’ll find even better versions of this technology, but tracking speeds are nowhere near real time yet. Here’s a particularly impressive one:

    • Ben Lang

      Great idea to use blood flow, though it would necessitate an extremely emotionally connected to actually get the player mad to detect it!

      The link you posted is really impressive, are you sure it isn’t in real time? It looks like they might be using a high-resolution depth-camera, or at least a camera better than a regular webcam!

      • John Dewar

        They’re using an HMC (Head-Mounted Camera) to capture the face. It’s got a row of lights arranged in an arc, with a camera in the center pointed at the face. The processing is fast but it’s definitely not real time, about 12 fps, with a source of 120 fps, which means it’s got to get 5-10x as fast to work – but there’s also the issue that we are glossing over which is that is going to be using up most of your processing power that you also need for a crazy-fast rendering loop! Here is a link with more videos of the tech: