Packing a 7-megapixel front-facing depth camera, Apple’s iPhone X can do some pretty impressive things with its facial recognition capabilities. While unlocking your phone and embodying an AR poop emoji are fun features though, the developers at Kite & Lightning just published a video of an interesting experiment that aims to use the iPhone X as a “cheap and fast” facial motion capture camera for VR game development.

Created by Kite & Lightning dev Cory Strassburger, the video uses one of the studio’s Bebylon Battle Royale characters (work in progress) to demonstrate just how robust a capture the iPhone X can provide. Flexing through several facial movements, replete with hammy New York(ish) accent, Strassburger shows off some silly sneers and a few cheeky smiles that really show the potential for capturing expressive facial movement.

While still a quick first test, Strassburger says that even though the iPhone X can drive a character’s blendshapes at 60fps while it tracks 52 motion groups across the face, “there’s a bit more to be done before I hit the quality ceiling in regards to the captured data.”

On the docket before the iPhone X’s TrueDepth camera can be levied as a VR game development workhorse, Strassburger says his next steps will include getting the eyes properly tracked, figure out why blinking causes the whole head to jitter, re-sculpting some of the blend shapes from the Beby rig to be better suited for this setup, visually tune characters, and add features to record the data.

SEE ALSO
Kite & Lightning's 'Bebylon Battle Royale' Nabs Unreal Dev Grant
image courtesy Kite & Lightning

To top it off, Strassburger is thinking about creating a harness system to mount the iPhone into a mocap helmet so both face and body (with the help of a mocap suit) can be recorded simultaneously.

Bebylon Battle Royale, a comedic multiplayer arena brawler, is due out sometime in 2018 on Rift and Vive via Steam. We can’t wait to see what the devs have come up with, as the game already promises to be one of the silliest games in VR.

This article may contain affiliate links. If you click an affiliate link and buy a product we may receive a small commission which helps support the publication. See here for more information.


  • This is great for animators. We already have IKinema mocap for body mocap using a Vive and now this for facial capture. This technology used to cost thousands at motion capture studios. For animators this will be vastly quicker than manual keyframing and give you full control to do retakes etc.

    I hope that Kite & Lighting get in touch with IKinema (https://ikinema.com/orion) and see if they can join forces?

    • Lucidfeuer

      iKinema seems a bit expensive and not that precise, as well as the set-up, bulky. Is it compatible with other motion suits or markers?

      • 400 pounds a year is great for full body (not fingers) capture. I recently did a ton of character animations and if I had this I would have done it all in a few days rather than 2 weeks. I also tested out one of their animation mo cap files in Unity and after some retargeting for the feet it looked great. Apparently is was raw unedited mocap too. It outputs BVH and FBX rigs/data so you can drop it straight onto your own unity model or import it into 3DSMax and apply to a biped.

  • dk
    • Lucidfeuer

      Tough Facerig is proprietary and ARKit/Core are open apis.

  • Thoreau

    Hilarious Bentley B… I mean Cary Strassburger