Hand-tracking Text Input System From Facebook Researchers Throws Out the Keyboard (sort of)


A prototype from Facebook Reality Labs researchers demonstrates a novel method for text input with controllerless hand-tracking. The system treats the hands and fingers as a sort of predictive keyboard which uses pinching gestures to select groups of letters.

Text input is crucial to many productivity tasks and it’s something which is still a challenge inside of AR and VR headsets. Yes, you can sit in front of a keyboard, but with a VR headset on you won’t be able to see the keyboard itself. For some very good typists, this isn’t an issue, but for most people it makes typing especially challenging. Even for good typists (or for AR headsets where the keyboard is visible), the need to sit in front of a keyboard keeps you chained to a desk, drastically reducing the freedom that you’d otherwise have with a fully tracked headset.

Voice input is one option, but problematic for several reasons. For one, it lacks discretion and privacy—anyone standing near you would not only have to hear you talk, but they would also hear the entire contents of your input. Another issue is that dictation is a somewhat different mode of thought than typing, and not as well suited for many common writing tasks.

A virtual keyboard in ‘Facebook Spaces’

Virtual keyboards are another option—where you use your fingers to poke at floating keys—but they’re too slow for serious writing tasks and lack physical feedback.

Facebook Reveals Latest Wrist-worn Prototypes for XR Input & Haptics

Facebook Reality Labs researchers have created a hand-tracking text input prototype, designed for AR and VR headsets, which throws out the keyboard as we know it.

Instead of touching keys on the keyboard, groups of keys are mapped to each finger. Instead of selecting a specific letter, you pinch with the finger corresponding to whichever color-coded group contains the desired key. As you go, the system attempts to predict which word you want based on context, similar to a mobile swiping keyboard. The researchers call the system PinchType.

PinchType overcomes many of the issues with typical virtual keyboards and voice input. It’s quiet, private, and looks to be much faster than hunt-and-peck on a floating virtual keyboard. It also provides feedback because you can feel when you touch your fingers together.

The researchers shared some initial findings from testing the system:

In a preliminary study with 14 participants, we investigated PinchType’s speed and accuracy on initial use, as well as its physical comfort relative to a mid-air keyboard. After entering 40 phrases, most people reported that PinchType was more comfortable than the mid-air keyboard. Most participants reached a mean speed of 12.54 WPM, or 20.07 WPM without the time spent correcting errors. This compares favorably to other thumb-to-finger virtual text entry methods.

But there’s some downsides. The system relies on accurate hand-tracking, and one of the most challenging facets of it—as seen from a head-mounted camera, it’s very common for fingers to be occluded by the back of the hand. Below, you can see that—as seen from the viewpoint—it’s ambiguous if the user is using their pinky or ring finger for the tap.

Nintendo Partners With Niantic for Multiple New AR Games, Starting This Year with 'Pikmin'

It’s very likely that the PinchType prototype was developed using high-end hand-tracking tech with external cameras (to remove sub-par accuracy from the equation). We’ll have to wait for the full details of the system to be published to know if the researchers believe these occluded cases present an issue for an inside-out hand-tracking system.

The PinchType prototype is the work of Facebook Reality Labs researchers Jacqui Fashimpaur, Kenrick Kin, and Matt Longest. The work was presented under the title Text Entry for Virtual and Augmented Reality Using Comfortable Thumb to Fingertip Pinches.

The work was published as part of CHI 2020, a conference focused on human-computer interaction.

This article may contain affiliate links. If you click an affiliate link and buy a product we may receive a small commission which helps support the publication. See here for more information.

  • Greyl

    Now all we need is a Phantasy Star Online-like VR MMO to make use of it.

  • Typing in VR is interesting even for touch typists like myself. I use a Logitech wireless keyboard with touchpad which is super easy to use in VR with Index controllers, I can hold the keyboard and type/trackpad the cursor whilst wearing the controller.

    But this is far from being a solved problem, as we move away from desktop do we even need QWERTY?


  • Shem

    The pinky stays up when you use your ring finger to pinch.

  • rbs

    I’d like to see an API side (available everywhere) tunable chorded keyboard using current or improved VR controllers. And a typing game to go with it, because there is a learning curve.

  • Regis Souza

    You can solve this making the user flip his hand palm to the camera. Very cool idea.

  • lnpilot

    That looks like one of the most excruciatingly tedious ways to “type”.

  • Camisade

    Still doesn’t feel quite as useful (or fast!) as simply being able to open a pass-thru “window” in the bottom of your VR “view” so you can see the keyboard. Being able to type quickly and accurately is critical for significant adoption (and especially in online games, arguably the market entry point for most people in VR, where you otherwise risk “death by typing” if it takes you too long to communicate (or enter commands) via text, which is common in games that support both flat-screen and VR play). VR headsets with external cameras need to better enable a hotkey or controller key press to allow someone to “see” the keyboard and hand positioning for fast typing without having to feel out where the home keys are.

    Any solution that has as a feature “10-15 accurate WPM, *after* a learning curve is dead in the water upon launch. That’s simply TOO slow. Better to enable us to better use the tech that’s second nature than to reinvent the wheel …and make it square.

    Yes, with pass-thru you’re still chained to your desk, but at least you can knock out text entry quickly, then minimize/remove that pass-thru window and get back to what you were doing.

    The only promising alternative I’ve seen thus far are the devices that attempt to map motions/nerve impulses to enable “typing” on a virtual keyboard based on finger movement (and their accuracy is way too low, still). Perhaps something like that, combined with this FB tech, could work together to provide greater functionality without having to learn a specialized skill that’s useful in only one (part-time) environment.

  • Nathanael Mooth

    I played around with a similar concept a while back with the leap motion and a VR headset. It was finicky, but you could get it working by turning your hands towards the camera. It was cool because your hand provides you the haptic feedback you crave while typing. However the solution didn’t strike me as robust enough for doing things like coding in VR since that is a situation where predictive models aren’t terribly good. Personally I think the solution lies in controllers similar to the knuckles that support multi-finger input. You can use the thumbstick to simulate a finger’s reach direction allowing each finger to reach a specific key similar to how they would on a keyboard. Per finger sensing still needs a bit more improvement but promises a way to eliminate the need for a predictive model.

  • Pablo C

    Make a good music game with this mechanic, we´ll be trained in no time.