Grabbing objects in VR may be one of the medium’s most fundamental interactions, but making it work well isn’t as easy as you might think. Developers often need to spend time hand animating the hand model so that it appears to hold each object in a realistic way. Developer Luca Mefisto has built a smart tool which uses Quest’s hand-tracking to enable developers to motion-capture hand-poses, making the whole ordeal quicker, more realistic, and ultimately more immersive for players.

Update (July 20th, 2020): Developer Luca Mefisto has released the first version of his HandPosing tool which uses Oculus Quest hand-tracking to quickly author realistic hand-poses for virtual reality interactions. The tool is available on GitHub.

“This is a work in progress, and things are subject to change. I hope it serves others either as a useful tool or at least as a starting point for their grabbing-interaction implementations,” Mefisto writes.

The original article below, which outlines the benefits and functions of the HandPosing tool, continues below.

Original Article (July 7th, 2020): Some VR games employ various methods of ‘dynamic’ animation to create realistic hand poses when players grab objects in VR (Lone Echo, for instance). Generally that’s proprietary tech, which means any developers wanting to do the same would need to build a similar system from scratch (not an easy task).

Rather than do that, some games cut out the hand-posing problem entirely by simply making your virtual hands disappear when you grab objects (Vacation Simulator, for instance).

Developers that want to keep the player’s hands visible need to create hand-poses manually so that when you grab an object, your virtual hand grips the object in a realistic way. It’s not that this is a difficult task per se, but it can be immensely time consuming.

At minimum you need one custom hand-pose for every uniquely shaped object in a given game. Even then, consider how many different ways players might want to hold a single object… even if you cut out unlikely poses, you still may need four or five poses for a single object to cover the most obvious grips. If there’s 100 uniquely shaped objects in a game, that could mean animating 400 or 500 hand-poses.

Three Totally Creative Uses of Quest Hand-tracking

VR developer Luca Mefisto wants to make this whole process quicker and easier—allowing developers to make more realistic poses in less time. He’s building a tool which smartly leverages Quest’s hand-tracking feature to allow developers to take a ‘snapshot’ of their own of their own hand gripped around virtual objects.

The tool then allows developers to define valid positions for the pose, allowing the hand to snap realistically to the nearest valid position on the object.

Objects can also have multiple poses and grabbing points to cover different ways of grabbing the same object (like the scissors below).

Though the tool requires Quest’s hand-tracking for creating the poses, Mefisto says the hand-pose tool will work for games that employ hand-tracking or controllers.

Though the tool is so far unnamed, the developer plans to release it as an open source project to the VR development community. You can follow Mefisto on Twitter to see updates on the tool’s development and keep an eye out for its release.

Alternative Text

This article may contain affiliate links. If you click an affiliate link and buy a product we may receive a small commission which helps support the publication. More information.

Ben is the world's most senior professional analyst solely dedicated to the XR industry, having founded Road to VR in 2011—a year before the Oculus Kickstarter sparked a resurgence that led to the modern XR landscape. He has authored more than 3,000 articles chronicling the evolution of the XR industry over more than a decade. With that unique perspective, Ben has been consistently recognized as one of the most influential voices in XR, giving keynotes and joining panel and podcast discussions at key industry events. He is a self-described "journalist and analyst, not evangelist."
  • Ad

    This is absolutely excellent and he deserves an award. This should have happened much sooner with leap-motions being in VR for like five years, and in VR editing software being a thing but having enough functionality. I hope he does add support for the Microsoft mixed reality toolkit if possible, so it can work with leap motions, Hololens, etc.

    • Is hand tracking possible with adaption of his software and with these cameras? They’re really not very useful at the moment, but I’m not a coder so no idea how difficult to implement.

      • Ad

        In theory yes, HTC got hand tracking working with two Vive pro cameras, but more useful would be using a leap motion, which fits easily in the frunk, although I mount mine under the frunk facing down slightly so I can more naturally use my hands.

        • Thanks for your reply. I thought the ultraleap module would make more sense.

        • Marc-André Désilets

          This software is made for running on the quest and uses oculus sdk features. You could make a similar software using the leap motion but I doubt that you’ll be able to use this one with it. Doing hand poses with leapmotion is easy, and you can even record animation using a motion builder plugin. You just wont have it in vr. The oculus quest is the only vr headset with hand tracking support so if you really need this features I think it would me smarter to simply buy a quest, the time you’ll save on your production will pay for it anyway.

          • Ad

            That… makes no sense. The developer says explicitly that there is no reason this couldn’t work with other hand trackers, no reason it has to only be used with VR, and you sound like you’ve never used a leap motion and don’t know it works in VR.

          • marcandrdsilets

            Sorry for being an ignorant fool. Message me when you find a way of making ‘this app’ work in vr using your leap motion, can’t wait to see it!

          • Ad

            It’s open source and according to the person who made it, it is something someone can absolutely do. I don’t understand what is so strange about a leap motion in VR. There has been a leap motion VR SDK and api for years now, it was a major push by them and there are dozens of VR programs that support it from flight simulators to commercial creative software like gravity sketch.

          • marcandrdsilets

            I’m not saying that it’s not possible to do vr hand tracking with the leap motion, I know it was the first hand tracking solution available for vr. And no it is not a standard yet, and other than techdemo I’ve never seen in officialy integrated to any games. What I’m saying is that ‘this tool’ requires Quest’s hand-tracking for creating the poses. I’m not saying hand poses requires a Quest. Sure you can play with the code and adapt it to use the leap motion but it will require software development, you will need a programmer or at least some coding knowledge, it’s not going to be plug and play. That’s all I’m saying.

          • Ad

            Obviously, but considering how useful this is, it would make sense to do so. This app isn’t plug and play right now, it’s not finished and he said there are maintenance issues. But any software engineer who understands the leap platform could work it in relatively easily.

            And yes, there are multiple games and programs that integrate the Leap in VR, probably more than the Quest still.

  • dmbfk

    “per say”? Is that the dipshit spelling of the Latin per se?

    • benz145

      Yes it is, and that dipshit is me. I’ll fix it, thanks.

    • Marc-André Désilets

      Instead of insulting people here, you can always get your news from another site, I don’t think we’re going to miss you much.

      • dmbfk

        Marc-André, long time no see. Still got halitosis?

    • Trenix

      Think the real dipshit is the one that gets butthurt over spelling.

      • dmbfk

        Have you got the teacher’s dick in your mouth again Trenix? You don’t need spelling when you’ve got your special oral talent in life I guess. Ooh please boss, I need a promotion! yom, yom, yom.. here you go have a car too, same time next week in the stationery cupboard? Fucking teacher’s pet.

        • Trenix

          You watch some weird pornos. You sound lonely.

          • dmbfk

            Dude that’s the worst pick up line I’ve ever heard.

          • dmbfk

            The only weird pornos I watch are the ones of your Mom getting gang-banged by retards. Apparently it was filmed 9 months before you were born.

          • Trenix

            Sound like a high school. Covid got you out of school I assume.

          • dmbfk

            What an awkward segway.

  • Marc-André Désilets

    Very interesting, it’s cool that you cover this kind of developer tools.

  • That’s an amazing idea actually! And it could be very useful for a game we are developing…

  • Lucidfeuer

    This is a smart and excellent idea. As always small (or median like R.A.D.) studios are doing the great job that mediocre corporations (per capita) don’t.