A newly published patent application from Apple highlights the company’s internal exploration of “micro-gestures” for AR input, which involves using the thumb against the index finger as a sort of virtual joystick or selector. Another newly granted patent covers the use of head-worn haptics as a means of directing the user’s attention toward virtual objects which are out of sight.

Alongside a string of acquisitions and job hiring focused on XR, Apple has long been exploring the space through internal R&D. While much of that work is secret, the details that emerge through the company’s patents shed light on some of the technologies and concepts Apple has been exploring in VR and AR.

In a patent application filed September 2020 and published this month, Apple covers the use of “micro-gestures” as a means of controlling XR devices. One version of this micro-gesture input envisions the user using their thumb, pressed against the side of their index finger, as a sort of virtual joystick for making directional selections.

Image courtesy Apple

The patent application indicates that Apple thinks this approach would allow for a hands-free, highly motor efficient input which is also unobtrusive:

“Using the cameras to capture the micro-gestures to interact with the three-dimensional environment allow the user to freely move about the physical environment without be encumbered by physical input equipment, which allows the user to explore the three-dimensional environment more naturally and efficiently,” the application states. “In addition, micro-gestures are discrete and unobtrusive, and are suitable for interactions that may occur in public and/or require decorum.”

The patent application shows several examples beyond the ‘virtual thumbstick’, including gestures which treat each segment of each finger as its own ‘button’ for input (ie: pressing your thumb to the middle section of your ring finger).

SEE ALSO
Hide-and-Seek VR Shooter 'Mannequin' Launches on Quest & Steam, Trailer Here
Image courtesy Apple

The patent also throws some serious shade at existing input modalities for XR devices:

“For example, systems that provide insufficient feedback for performing actions associated with virtual objects, systems that require a series of inputs to achieve a desired outcome in an augmented reality environment, and systems in which manipulation of virtual objects are complex, tedious and error-prone, create a significant cognitive burden on a user, and detract from the experience with the virtual/augmented reality environment. In addition, these methods take longer than necessary, thereby wasting energy. This latter consideration is particularly important in battery-operated devices.”

Interestingly, Facebook has shown essentially the same exact ‘virtual thumbstick’ as an input concept used with its prototype wrist-worn XR input device:

This particular patent application hasn’t yet been granted to Apple yet, but something tells me the two companies are likely to be butting heads in the near future over XR patents.

Another Apple patent, this one filed November 2018 and granted last month, envisions head-worn haptic devices which would be used to help direct the user’s attention to objects that are outside of their field of view.

Image courtesy Apple

The patent envisions three different levels of head-worn haptics, ranging from those implanted in earbuds or glasses all the way to a headband forming a complete circle of haptic modules.

Image courtesy Apple

The haptics would be used, in some way, to turn the user’s attention to one side or the other in order to draw their gaze toward something. Use-cases spelled out in the patent include helping the user locate an out of view speaker during a VR or AR teleconference, or directing the user’s attention to an important object inside of a virtual reality environment.

SEE ALSO
Apple Reportedly Rethinking XR Headset Approach Amid Reveal of Meta's AR Glasses
Image courtesy Apple

Exactly how haptics would be superior for guiding the user’s attention over simply using spatial audio isn’t entirely clear from the patent.

We’ve seen head-worn haptics experimented with previously, though for an entirely different use-case; researchers found they could significantly reduce discomfort related to smooth locomotion in VR with haptics mounted on a headset.

Newsletter graphic

This article may contain affiliate links. If you click an affiliate link and buy a product we may receive a small commission which helps support the publication. More information.


Ben is the world's most senior professional analyst solely dedicated to the XR industry, having founded Road to VR in 2011—a year before the Oculus Kickstarter sparked a resurgence that led to the modern XR landscape. He has authored more than 3,000 articles chronicling the evolution of the XR industry over more than a decade. With that unique perspective, Ben has been consistently recognized as one of the most influential voices in XR, giving keynotes and joining panel and podcast discussions at key industry events. He is a self-described "journalist and analyst, not evangelist."
  • TechPassion

    They want to patent human movement? Fckng company. They showed nothing till 2021 and only spy on others, patent troll. After Steve died, this company struggles to present anyhing valuable. Cook is ok to manage the company, but zero as inventor.

    • mirak

      they would patent walking

    • sebrk

      Oh yeah they really “struggle”. LMAO.

      • Frankie Hanlon

        Amara this-past month I collected $19562 thru working on pc at home in my spare time..(s1189) I figured out to do it by spending some hours in whole day consistently computer.(s1189) it’s very nice and any of you definetly can get this. >>> https://ulvis.net/PzZ

      • Cheryl Johnson

        Get $193 per h from Google!…~a1232~ Yes this can be best since I simply got my initial payroll check of $24413 and this was just of one week…I am aslo purchased my Mclaren P1 right after this payment…~a1232~ it is really best job I have even had and you will not for~give yourself if you not check it >>>> http://www.riverbridge.cf/check2020 !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!

    • Andrew Jakobs

      It wasn’t just after Steve died, it happened long before that.

  • Microgestures are a nice idea… I think in the end all companies will adopt them. They’re discrete, effortless, and easy to learn

  • Haptics for guiding the user’s attention would be useful for deaf people.