Some of the most basic questions surrounding AR/VR tech aren’t entirely solved yet, like making text input a fast, comfortable, and familiar experience. Facebook’s Reality Labs (FRL) today revealed new research into hand tracking which aims to bring touch typing to AR/VR users, all without the need of a physical keyboard.

VR keyboards haven’t evolved beyond this, Image courtesy Virtual Desktop

There’s already basic hand tracking on Quest which lets you navigate system UI, browse the web, and play supported games like Waltz of the Wizard: Extended Edition (2019) without the need of Touch controllers, instead letting you reach out with your own to hands to cast spells and manipulate objects.

As interesting and useful those use cases may be, we’re still very much in the infancy of hand tracking and its potential uses for virtual reality. Using your fingers as glorified laser pointers on a virtual keyboard reveals just how much of a gap there is left in natural VR input methods. On that note, Facebook researchers have been trying to build out hand tracking to even more useful applications, and their most recent is aimed at solving some of the most frustrating things in VR/AR headset users to this day: text input.

Facebook today revealed that its FRL researchers used a motion model to predict what people intended to type despite the erratic motion of typing on a flat surface. The company says their tech can isolate individual fingers and their trajectories as they reach for keys—information that simply doesn’t exist on touch screen devices like smartphones and tablets.

“This new approach uses hand motion from a marker-based hand tracking system as input and decodes the motion directly into the text they intended to type,” FRL says. “While still early in the research phase, this exploration illustrates the potential of hand tracking for productivity scenarios, like faster typing on any surface.”

‘Township Tale’ Studio Reveals Glimpse of Next Project, a “dark fantasy VR game”

One of the biggest barriers to overcome was “erratic” typing patterns. And without the benefit of haptic feedback, researchers looked to other predictive fields in AI to tackle the issue of guessing where fingers would logically go next. FRL says it researchers borrowed statistical decoding techniques from automatic speech recognition, essentially replacing phonemes for hand motion in order to predict keystrokes—that’s the short of it anyway.

“This, along with a language model, predicts what people intended to type despite ambiguous hand motion. Using this new method, typists averaged 73 words per minute with a 2.4% uncorrected error rate using their hands, a flat surface, and nothing else, achieving similar speed and accuracy to the same typist on a physical keyboard,” the researchers say.

The research on typing with hands was conducted using a high-precision external hand-tracking system. Likely the researchers wanted to prove the idea that hand movements alone could be used to reliably predict intended typing before trying to adapt the system for less accurate hand-tracking data like what Quest and Quest 2 are capable of. That said, hand-tracking on Quest is an ongoing area of research and it’s likely to improve with time. Below we can see how Facebook has been able to contend with tracking a single hand across multiple camera views and seamlessly transitioning between them:

With its insights into hand tracking, Facebook is undoubtedly preparing for the next generation of AR headsets—the ‘always on’ sort of standalone AR headsets that you might wear in the car, at work, at home and only take off when it’s time to recharge. Using Quest 2 as a test bed for AR interactions sounds like a logical step, and although the company hasn’t said as much, we’re hoping to see even more cool hand tracking tech pushed out for experimental use on the new, more powerful standalone VR headset.

Newsletter graphic

This article may contain affiliate links. If you click an affiliate link and buy a product we may receive a small commission which helps support the publication. More information.

Well before the first modern XR products hit the market, Scott recognized the potential of the technology and set out to understand and document its growth. He has been professionally reporting on the space for nearly a decade as Editor at Road to VR, authoring more than 3,500 articles on the topic. Scott brings that seasoned insight to his reporting from major industry events across the globe.
  • Rawbot

    Typo. phenomes must be phonemes

  • Tyler W

    FRL is moving forward at light speed with vr innovation. Its pretty exciting, but it seems everyone else is getting left behind, and I am not sure I trust Facebook so much anymore.

    • Ad

      They really aren’t. They just publish their research with marketing copy. HTC, Google, Sony, Valve, and others have a lot of the same research but it just doesn’t get publicly released or it gets ignored. Notice how FRL’s DFR news died out, it’s because the kind of extreme DFR they were talking about is impossible without being incredibly distracting or taking away essential information.

  • Bumpy

    Why are we touch typing in VR? Shouldn’t we be simply dictating and tell computers what to do instead.

    • alxslr

      For almost the same reasons for which we don’t simply dictate and tell PCs what to do.

  • Great research, but very few people know how to touchtype… so this would be useless for most people out there… also I guess that every person touchtypes in a different way, so the system should be calibrated for the user

    • Just because it shows typing without a physical keyboard doesn’t say the user can’t see a virtual keyboard in VR…

    • Ad

      I think most people who type as a major part of their work can touch type, but since you can’t do this standing, I think being able to use your phone keyboard in VR would be ideal.

    • wheeler

      Also, since it requires a surface anyway … you may as well use a keyboard and actually have decent feedback. Feedback is such a critical part of keyboard interaction

      • Tony V.

        The brain can adapt by relying more on proprioception. I suspect it has the potential to be faster too because it doesn’t require receptors on the skin to relay tactile feedback (adding a few milliseconds?). It could be improved by using reinforcement learning AI to correct for small inaccuracies based on the times you are using the backspace button. There is no reason to reach all the way to a button, over time it could learn to infer the button by a small wiggle of the fingers. Curious to see if they take it that far.

  • Azreal42

    Well, that’s great but right now, hand tracking on oculus is unusable when lying and almost unusable when sittings, so …

  • Ad

    “without the need of Touch controllers”

    We need a style guide that doesn’t use language like this.

    I really hope Valve, WMR, and HTC get ahead of them on the AR side of things. It’s terrible quality for AR and the competition get run circles around them if they actually try.

    • Captain Comedy

      responses to this are so pessimistic.

      I can type with no keyboard, table or vr and know I will have a certain level of accuracy solely based on muscle memory. what they are trying to do is entirely achievable, but they want a 100% viable solution and not a half cooked one – completely understandable.

      You will never be an investor with that attitude.

      • Ad

        What I’m saying is that there is no such thing as software that doesn’t require VR controllers. Controllers add so much to the experience that is never being fully replaced by hand tracking. It’s not comfort either because you can let go of knuckles but they never get the abstract praise that hand tracking does. I have a leap, I know that hand tracking is incredibly limited and will never lose half those limitations.

        • care package

          My favorite VR experience to date is still Alien Isolation which didn’t use VR controllers. VR controllers do add to immersion and they’re great for gamercising, but really it’s whats over your face that makes the biggest difference.

          • Ad

            1, it doesn’t use hand tracking so sure. And 2, that game is nice, but for me it isn’t the face that makes the biggest difference.

  • dk

    what’s that black thing on her face

    • Ad

      OG Rift

  • dk
  • Patrick Hogenboom

    Looks impressive.
    But I think that swipe typing would be more suitable, and way easier to implement for VR text entry.
    Especially when you ignore aligning the virtual keyboard and physical surface (as they do in this article)

  • Smartphone typing was revolutionized by, first, T9, and second, when fullsize keyboards were a thing on displays, swiping. Use swiping technology on the virtual keyboard by pinching to type and stop pinching when you want a space … maybe this will catch on when Apple sets swiping as default on iPhones? :|

    • Bonus, you will feel like a principal conductor of an orchestra!

  • TechPassion

    What is the problem to paint the keyboard in VR so people could touch the keys virtually.