Oculus Claims Breakthrough in Hand-tracking Accuracy


One of Facebook’s underlying goals for VR is to use it as a means of connecting distant people. While today friends can talk and play in a range of social VR applications, including Facebook Spaces, the representation of users in VR is still a caricature at best. At F8 this week, Oculus showed work being done on hand-tracking to bring more intuitive control and accurate avatars into VR.

Maria Fernandez Guajardo, Head of Product Management on Core Tech at Oculus, this week shared glimpses of her work, which she describes as turning the company’s various research projects into practical technologies for future products. Beyond hand-tracking, she also revealed the Oculus ‘Half Dome’ prototype, a headset with a 140 degree field of view and varifocal display.

Guajardo detailed a computer-vision based hand-tracking system trained with a self-optimizing machine learning algorithm, which she says achieves tracking that’s “far more accurate than any method before for tracking a single hand, two hands, and hand-object interactions.” Footage which appeared to show the hand-tracking in action (above) also appeared to show detection of snapping gestures.

Image courtesy Oculus

Guajardo explained that the company used a marker-based tracking system to record hand interactions in high fidelity, and then condensed the recorded data into 2D imagery which allowed them to set a convolutional neural network to the task of uniquely identifying the positions of the markers across a large set of hand pose imagery, effectively allowing the system to learn what a hand should look like given an arbitrary set of marker positions. Ostensibly, this trained system can then be fed markerless camera input of a user’s hands and solve for their position.

Image courtesy Oculus

By measure of something Oculus labeled ‘Tracking Success Rate’ (and also described as “accuracy,” the company claims to have achieved a rather astounding 100% success rate with single hand-tracking, up from the claimed 90.49% of other hand-tracking methods. The company claims even bigger leaps compared to other methods for two-handed and hand-object interactions.

Oculus Reveals 140 Degree VR Headset Prototype with Varifocal Displays

Hand-tracking can enable rich, controller-less input, which could be especially suitable for mobile VR headsets where carrying separate controllers would hinder portability. While the accuracy achieved with Oculus’ approach is certainly impressive, it isn’t clear what the computational constraints are at this stage.

This article may contain affiliate links. If you click an affiliate link and buy a product we may receive a small commission which helps support the publication. See here for more information.

Ben is the world's most senior professional analyst solely dedicated to the XR industry, having founded Road to VR in 2011—a year before the Oculus Kickstarter sparked a resurgence that led to the modern XR landscape. He has authored more than 3,000 articles chronicling the evolution of the XR industry over more than a decade. With that unique perspective, Ben has been consistently recognized as one of the most influential voices in XR, giving keynotes and joining panel and podcast discussions at key industry events. He is a self-described "journalist and analyst, not evangelist."
  • jean thompson

    Damn.. that’s looks awesome.

  • iThinkMyCatIsAFlea

    Of course those arms are white! Google Oculus Trump.

    • Snooder56

      Actually they are translucent you racist piece of shit. The only reason racism in this country exists the way it does is because petty fucks like you keep bringing it up!

      • Get Schwifty!

        Don’t forget you have a block user feature… it’s useful for filtering out the immature, mentally challenged and just plain trollish.

        • iThinkMyCatIsAFlea

          …and the truth.

          • Andrew Jakobs

            But that’s not what you are telling.. oh or lefties never tell lies? oh wait… lefties are just as bad as righties…

      • iThinkMyCatIsAFlea

        Are you really that dumb?

        “White attacker is found guilty in the beating of a black man in a parking garage during Charlottesville protests”

        • Rob

          Wow one articial. There are so many black on white crimes that goes unsaid. Don’t try to start stuff. YOU ARE WHAT IS THE PROBLEM WITH THIS COUNTRY….PEOPLE THAT THINK LIKE YOU.

        • Andrew Jakobs

          So if a black man is beating a white man in a parking garage it’s also a racist thing? oh no, because according to people like you a coloured person can’t be a racist, only white people can. Ohhh, according to you I’ll propably even be a racist for even suggesting it. Please get your head out of your ass.. Yes, racism is a problem, but calling everything racist because a white person did something to a coloured person is just being ignorant..

      • Engineer_92

        Pretty sure the reason racism exists is because of racists

      • NooYawker

        His comment is dumb, but yours is out right moronic. Racism exists because people keep bringing it up??

      • iThinkMyCatIsAFlea

        Snooder56 “Actually they are translucent you racist piece of shit. The only reason racism in this country exists the way it does is because petty fucks like you keep bringing it up!”

        Quoted! There’s a very high possibility he’ll delete it.

        • Andrew Jakobs

          Why would he delete it, it’s true.. morons like you are screaming “racist” even though nothing racist has been going on here.. It would be racist if they deliberatly used a black womans arm because they want to not come off as racist (it’s the intent with what it’s being done).. The only racist here is you…

      • Pasi Ripari

        He’s not wrong… There’s still real racism left, but if you introduce anti-racism to an environment that has no racism, you actually end up making racism. Too much of anything, no matter what, is always a bad thing. Old saying, but keeps being forgotten.

    • Michael Abia

      Snooder56. don’t you see?, he thinks his cat is a flea.

    • Pasi Ripari

      Umm, why does it matter his arms are white?

      • iThinkMyCatIsAFlea

        Google Oculus alt-right.

        • Andrew Jakobs

          Oh please, not you again with this awful right-winged crap… go away..

          • wcalderini

            I’m not a racist, but I am a prejudiced bigot. I just can’t seemed to tolerate ignorant, flame baiting trolls. Maybe I should see a therapist or something. I just hate “stupid”. I’ll work on it.
            (Not directed at you AJ, my target should seem obvious)

        • Justos

          Mental illness can be treated, go seek help!

    • Ombra Alberto

      Troll Troll Trooll.. childen!!!

    • Garbagio Dumpsterino

      AND they’re a woman’s arms!!!

  • For serious software any advancements in natural hand gesture tracking is a great thing to see coming along.

    Games will always have the problem of occlusion and line of sight though, if it can’t see what you are doing with your hands/fingers then it wont be able to do anything meaningful.

    e.g. Grabbing a gun and putting your hands behind your back or to the extreme sides (think The Matrix lobby scene) . Your body or even your own hand/fingers could occlude gestures making it a special case use only.

    The VR Glove or VR thimble will emerge as the most accurate way to track hands/fingers no matter if the HMD does not have line of sight. Hopefully that will be sub mm accurate and low power too.

    • Mei Ling

      The main problem I see with any sort of full-on hand based tracking (with or without gloves) is how do you abstract your walking movements into the virtual world whilst sitting in place and not using any sort of leg tracking. Essentially how do you enable full “abstracted” locomotion when your hands are doing what they are supposed to be doing.

      If Oculus decide to release a version of their headset with hand gesture tracking then as far as I’m aware one hand would technically still need to hold a controller of some sort with a joystick and the other hand being free.

      • Well if gesture tracking works as it should then you could find a way to include simple teleport gesture positioning. But then you can’t be shooting a pair of guns at the same time or holding objects in both hands while moving around.

        Maybe a foot rocker pedal will be developed (like car pedals for seated experiences) or for ultimate immersion we would be using a slippery floor and a simplified version of the hand tracking but it tracks your your feet to allow both hands and both feet to be tracked / gesture tracked.

        • Mei Ling

          Valve Knuckles seems to be a sort of “best of both worlds” approach in that you strap the device onto your palm allowing your fingers to be free whilst having a track pad on your palm for gripping and abstracted functionality such as locomotion.

  • Cool

  • NooYawker

    They bought Leap Motion?

    • cartweet

      They bought NimbleVR which was similar tech years ago.

  • impurekind

    I’d like to see a system that uses only your hands for general menu interaction and stuff and then also lets you use a proper physical control when actually playing games of doing things where being able to grip something and feel tactile feedback is better. That would be a great dual solution.

    • Laurence Nairne

      Leap Motion handle this in their latest version of the Orion SDK though I’ve not worked on implementing it into any projects so couldn’t tell you how well it works. I also find that the 90.49% success rate of tracked positions supposedly achieved by their system is too prone to failure to be used in an actual product anyway.

      • Andrew Jakobs

        But don’t forget, it’s working with pretty old hardware and doing a pretty good job. They need to have new up-to-date hardware and I’ll think it will go up to 98% or something (100% tracking just isn’t possibe, not even a regular controller does 100%).

        • Laurence Nairne

          Aye it is old hardware, but computer vision is mostly a software challenge. It’s always about guesstimating finger positions on the information that the camera can see.

          New hardware won’t solve that. But you’re right, 100% is marketing speak.

  • sebrk

    Did I just smell Leap Motion fart their pants?

    • NooYawker

      Or preparing a law suit.

      • Justos

        You cant patent hand and finger tracking. (just the method of said tracking) Oculus bought nimbleVR and this is the fruits of their labour.

    • Andrew Jakobs

      Not really, as their tech doesn’t require expensive machinelearning and neural networks. LeapMotion is doing all this already with cheapass hardware, and even with their already ‘old’ hardware they are doing pretty good, really looking forward to what they can do with up to date hardware..

  • Stefan Küppers

    Hmm, all I see wiggling fingers and stuff leap motion showed ages ago. Maybe I am a cynic but please show me someone typing on a keyboard so I can be convinced that precision has improved over the years….

    • Bryan Ischo

      But these wiggling fingers appear to be much more accurate than Leap Motion’s wiggling fingers, which is worth talking about.

    • cartweet

      The difference here is that it’s training model can infer what your hand is doing even when your fingers are obscured. Leap motion does a pretty good job but their charts are showing with a single hand they can achieve 100% accuracy and a very high degree with hand-hand. As an example their system can identify when you snap your fingers.

  • Brad

    The problem with this kind of optical hand tracking is that it only works when the camera, presumably mounted on the headset, is looking at your hands, so if your hands are by your side, or you look away, it has no idea what your hands are doing.

  • Rob

    That is incredible, but unless they invent some kind of gloves, it won’t feel like you’re holding something…

  • flamaest

    100% , really? So it’s perfect and ready to ship, sure it is. God I hate this marketing BS.

    • Matt Clark

      It’s not hard to believe that the test they ran achieved 100% perfect accuracy. They never said it’s 100% accurate. That’s for stupid people to mistake and believe it’s perfect.

    • Da Mo (JFlash)

      But just look at the graphs, what more proof do you need! Graphs don’t lie

    • Raphael

      Rage kiddy gets insanely pissy because he wants it now or else it has to be bullshit.

  • brandon9271

    Cool.. so just incorporate this with inside out tracking cameras for a completely self contained, trackerless system!

    ….or better yet, sit on the tech for YEARS and let somebody else beat you to market!

    Yay, Oculus!

    • Lucidfeuer

      ahah that one made me laugh.

      • brandon9271

        Glad I could get a laugh :) The way things work in tech I’m afraid there’s more than a kernel of truth to my “joke”. I have a feeling Vive Pro wouldn’t have happened if things like Pimax 8k and Samsung Odyssey hadn’t showed up.

    • Toma Meneses

      Sounds like how Valve sat on their lazy asses for years with a fully functional VR system, and proceeded to let Oculus beat them to the 6dof market.

      Valve could have released Vive a full year ahead of Rift if they put some “give a damn” in their products. The only reason Valve even got around to launching it was because of the $2B wakeup call to Gabe that was FB’s acquisition of Oculus.

  • Mike

    I just came home from a bar and opened this, and misread the first line as “One of Facebook’s underlying goals for VR is to use it as a means of constantly dissing people.”

  • Rolf Adelsberger

    Optical hand tracking certainly offers ease of use. However, there are multiple problems that you cannot solve with a lightweight setup. Haptics, for example. Or occlusions or even self-occlusions…

    Have a look here: https://www.youtube.com/watch?v=Aej-wrgHq6Y

  • Meh

    I’m holding a golden torch for Microsoft’s Mixed Reality, when I picked up a Lenovo headset zero vr research went into my purchase. I couldn’t be happier, I know no other technology at the moment has hand tracking “yet” but until we see software that actually exploits it this makes it a sort of useless feature. Furthermore I think hand tracking could probably work on WMR more effectively than other vr gear, this is because of Microsoft’s inside out cameras that are built into the headsets. Take that with a grain of salt though, currently WMR does not support hand gestures. What I’m trying to say is I can see it working flawlessly in some future update.

  • Lucidfeuer

    Every time I read “neural network” now, my brain skips straight to “vaporware”. So how much value did these vaporpatents bring to Facebook stocks?

    Because of course adding 20cent of the required implementation and manufacturing cost to every headsets is too expensive to integrate this hand-tracking tech in the next Oculus…so yeah vaporware bullshit as always.

    • Raphael

      Irrelevant. The point is great strides are being made in natural control systems and YES it does eventually become consumer ready. Why are you such a miserable cunt by the way?

      • Lucidfeuer

        Yes, the billions of stock valuation that is running corporate strategy is “irrelevant”. Why are you such an hypocrite loser by the way?

        • Raphael

          I was born that way flappy?

          • Lucidfeuer

            one is not born a way, one becomes it.

  • brandon9271

    This will never be a 100% accurate solution because there’s always going to be some position where your fingers get occluded by part of your hand. Probably good enough for most applications but not for anything super accurate like typing.

  • Jason Gorman

    The key here is the CNN (convolutional neural network). We are going to see these kinds of huge advancements popping up everywhere in the next few years. Those who think this is not possible, I think haven’t quite wrapped their heads around what neural networks are capable of. A quick trip github and typing in VR CNN or VR Neural Network will give you a glimpse.

  • DanCarmon

    And only 2 years after Intel achived and showed at CES real hands tracking! Yay facebook!


  • Rolf Adelsberger

    I already posted it earlier: optical tracking is fine for specific applications. However, as these systems don’t actually measure, but rather “match images” to pre-learned gestures, they will not be able to detect arbitrary gestures. Especially in situations where the cameras cannot see the fingers anymore…

    Have a look: https://youtu.be/Aej-wrgHq6Y