Oculus brought controllerless hand-tracking to Oculus Quest as an experimental feature back in December, and while it’s useful for navigating around the headset’s menus when you don’t want to pick up the controllers, developers have begun prototyping more creative uses for the tech.

Controllerless hand-tracking on Quest is a great convenience feature. While we still expect controllers to be the go-to for hardcore games, there’s definitely opportunity for casual and novel games and experiences to prosper with hand-tracking.

Daniel Beauchamp is one VR developer that’s dreaming up creative uses for hand-tracking on Quest that go far beyond laser pointer and touchscreen interactions. He’s created a series of prototypes as a means of understanding the capabilities of the feature and what kind of interactions and experiences can bring delight to users. Here’s three that he shared recently.

Thrown Hand

Turning the tracked hand into its own independent object allows for a ‘remote control’-like capability where the hand propels itself around the environment by ‘crawling’ as the player continues to move the fingers on their real hand. Intriguingly, one could consider this a form of VR locomotion which only applies to a specific body part. This also extends the range of potential interactions in a more interesting way than 2DOF interactions like laser pointers and guns.

Finger Skateboard

Harkening back to the ’90s fad of fingerboarding (AKA Tech Decks), Beauchamp envisions a game mechanic where players control a tiny skateboard with their fingers. In later experiment he even showed some basic physics, allowing the user to flip the board to do tricks.


Hand Walking

While it’s easy to get around using VR controllers thanks to a combination of 6DOF motion and physical inputs like buttons and sticks, things get more complex with just your hands. In this prototype, Beauchamp shows a locomotion scheme where the player puppeteers a leg gesture with their hands to walk forward. While not likely a serious solution for VR locomotion, it’s a thought-provoking concept and one which could form the basis of other interesting interactions involving the personification of the user’s hand.

Experimenting Outside of the Box

Beauchamp is the Head of VR and AR at Shopify, a leading e-commerce platform. While he told us that his Quest hand-tracking experiments are done on his personal time, he applies the same concepts to how he thinks about the intersection of XR and e-commerce.

“One of the best ways to unlock new & powerful ideas is to build upon silly ideas. That’s what I’m doing with hand-tracking. Rather than starting with ‘How can I build a useful product with hand tracking’, I play around with many little ideas that bring a smile to people’s faces. They may seem random, but I’m learning a lot about what’s possible with the tech and interactions that could be applied elsewhere,” Beauchamp said. “I wish more VR devs did this. Don’t put the burden on yourself to build out a whole game or build out a whole product. Build many small things, no matter how silly they may seem. You’ll be surprised at just how much you learn.”

Delighting Users with Rich Interactions is Key to Making VR Engaging & Effective

He stresses the need for rapid-prototyping, something he felt important enough that he and the Shopify VR team built a VR rapid-prototyping tool for rapid experimentation.

“A lot of people are surprised Shopify has a VR team. We’ve had one for the past five years. Our goal has been to figure out how VR will impact the world of commerce.
This problem space requires outside-the-box thinking. Every time I talk about VR & shopping, people immediately suggest shopping malls. And to me that’s such a narrow view,” said Beauchamp. “In VR you have the ability to do anything you want, be anyone you want, and be anywhere you want. You are a wizard…and you want to shop in a regular mall? Wizards don’t shop at boring malls. So we’ve tossed out a lot of the notions you’d expect from a brick and mortar store bound by reality. We’re looking at things that are delightful and effective that are now possible in this new medium.”

But how do you think outside-the-box if the box is all you can see? Beauchamp suggests tossing out what you think you know.

“I think you just have to consciously throw out the [interaction] tropes [of real life and traditional games]. Like just prevent yourself from using them. Come up with ideas that don’t use them first, then evaluate if you still need them.”

– – — – –

Quest hand-tracking is expected to eventually leave its experimental status behind and become a full-blown feature on Quest, and other VR headsets are likely to follow, making the time ripe for developer experimentation ahead of implementation.

Newsletter graphic

This article may contain affiliate links. If you click an affiliate link and buy a product we may receive a small commission which helps support the publication. More information.

Ben is the world's most senior professional analyst solely dedicated to the XR industry, having founded Road to VR in 2011—a year before the Oculus Kickstarter sparked a resurgence that led to the modern XR landscape. He has authored more than 3,000 articles chronicling the evolution of the XR industry over more than a decade. With that unique perspective, Ben has been consistently recognized as one of the most influential voices in XR, giving keynotes and joining panel and podcast discussions at key industry events. He is a self-described "journalist and analyst, not evangelist."
  • impurekind

    These are all rather cool. :)

    • sasha

      Very long office hours and still unable to attain the things you really deserve. A trip planned for a long time and still unable to go. Your office work doesn’t permit you to spend very much needed time with your beloved. For everybody who is experiencing all of this? In that case, we certainly have much needed solution available for you. A job opportunity which happens to be completely on line. It means you can work from any location for few hrs and can earn around $24000. It truly is a fantastic opportunity to be your own boss and live the way in your own terms . Look at this awesome job opportunity >>> earn9kpermonth.bilder.lc

  • guest

    Reminds me of Thing from the Adams Family! For those of you not old enough to remember check out the youtube Best of Thing: https://m.youtube.com/watch?v=JTEVBcmPzk4

    • Cousin IT

      That’s thinking outside the box. Way ahead of its time for AR Glasses…

  • I love the one where you can throw your hands! It is super original!