A video posted by a Vision Pro developer appears to show the current levels of hand-tracking and occlusion performance that Apple’s new headset is capable of.

Apple Vision Pro, expected to launch in the next few months, will use hand-tracking as its primary input method. While system-level interactions combine hand-tracking with eye-tracking for a ‘look and tap’ modality, developers can also build applications that allow users to interact directly with applications using their hands.

Hand-tracking on Vision Pro can be broken into two distinct capabilities: ‘hand-tracking’ and ‘hand-occlusion’.

Hand-tracking is the estimated 3D model of the hand, its joints, and the position of the fingertips. This model is used to determine when objects are touched, grasped, and interacted with.

Hand-occlusion deals with how the system overlays your real hand onto virtual content. Rather than drawing a 3D model of a virtual hand into the scene, Vision Pro cuts out the image of your real hands to show them in the scene instead. Doing so adds another layer of realism to the virtual content because you can see your own, unique hands.

A developer building for Vision Pro using Unity posted a video which gives a clear look at both hand-tracking and occlusion.

In the video above, the real hands show the occlusion system in action, while the green dots show the estimated 3D position of the hand.

We can see that hand-occlusion is very good but not perfect; when surrounded entirely by virtual content and moving quickly, you can see some clipping around the edges of the hand.

Meanwhile, the actual hand-tracking position lags further behind the hand-occlusion, somewhere around 100–200ms behind. We can’t quite determine the true latency of Vision Pro’s hand-tracking however, because we only have a view of the occlusion-tracking hands to compare it to (which themselves will have some latency with regard to the real hands).

In this context, the video comes from a Unity developer conversation saying that their early experiments with Vision Pro showed very high hand-tracking latency, apparently in comparison to the latest Quest hand-tracking capabilities. Other Unity developers agreed that they were seeing similar latency on their own Vision Pro devices.

SEE ALSO
The Most Anticipated VR Games of 2024 Coming to Quest, PSVR 2, and PC VR

However, the video in question is a hand-tracking integration inside of a Unity app, which means the hand-tracking performance and latency may have additional factors relating to Unity that aren’t present when using Apple’s first-party Vision Pro development tools. And considering the headset hasn’t launched yet and these tools are still in development, we may see additional improvements by the time the headset actually reaches stores.

Newsletter graphic

This article may contain affiliate links. If you click an affiliate link and buy a product we may receive a small commission which helps support the publication. More information.


Ben is the world's most senior professional analyst solely dedicated to the XR industry, having founded Road to VR in 2011—a year before the Oculus Kickstarter sparked a resurgence that led to the modern XR landscape. He has authored more than 3,000 articles chronicling the evolution of the XR industry over more than a decade. With that unique perspective, Ben has been consistently recognized as one of the most influential voices in XR, giving keynotes and joining panel and podcast discussions at key industry events. He is a self-described "journalist and analyst, not evangelist."
  • MackRogers

    Everyone has been waiting for the “Killer App” in VR. A DECADE we have been waiting for the killer app.

    It turns out the killer app was never a game. The killer app will be Apple Vision eye tracking, hand-tracking, paired with the microLED screens.

    Up until this moment we have been using a fucking LASER POINTER to navigate blurry menus to interact with a virtual environment. Do any of you here have any fucking clue how archaic and backward that is?

    I wish I could fast forward 30 years and laugh at how stupid and idiotic some of you are here that do not realize what we are on the cusp of.

    • ViRGiN

      Who you’re talking to?
      Noone claimed we are at the peak of possibilities lmao.

      • MackRogers

        I am talking to the overwhelming majority who think VR Gaming is the future and Apple Vision is trash because of its lack of gaming focus.

        To Deckard cope morons. To Meta plastic toy fanboys. BigScreen simps. PSVR2 apologists.

        Every single one of them believes VR should be gaming oriented, and every single one of them is why this industry is on the verge of collapse and is littered with the tombstones of hundreds of developers who have been burned investing in the space.

        • Mike

          I get what you mean. But the biggest obstacle to mainstream VR adoption in gaming (which is a very lucrative industry) is comfort. Most people don’t wanna be uncomfortable while relaxing with entertainment. Bigscreen is a huge step in the right direction.

          • MackRogers

            It’s easy to have an ultra light minimalistic design when you tether yourself to a desktop PC. Full off load of processing and power.

            I wish Apple would just make the battery pack 3x thicker and put the M2 inside it and get as much of that crap off your face as possible. But being fully hardwired tethered like Bigscreen is a stupid deadend parlor trick of a device.

          • VrSLuT

            What? At a dead end with a porn trick? I hear they are going to stretch the other end of their 1st Amendment rights so you can f*ck things rather than kill things in VR. Then maybe we won’t have as many guys starting wars in the world!

          • Mike

            Wouldn’t call it easy. And it’s a bit strange to call all tethered headsets “a parlor trick” when it’s been the standard for VR since the beginning. Standalone is relatively new, and is the minority of headsets.

            Of course wireless is nice, but it’s not the only important factor.

          • MackRogers

            Go ahead and tell/show me the roadmap for any other fully tethered headsets. That’s not happening anymore. It’s over. It will be fully wireless or on belt like AVP. So yea, it is a stupid parlor trick that has no legs. Unless they want to keep selling 1k-5k units a year.

          • ViRGiN

            Somnium, which likely won’t release at all, or will release in extremely limited numbers and flop immediatly, will be a tethered headset, from the makes of flopped metaverse “Somnium Space”. Great product for youtubers to masturbate about and drop affiliate links.

          • Mike

            Comfort is key. Cables are annoying, but you get used to them. Nothing compared to comfort. And graphics are very limited on standalone.

          • ViRGiN

            Relatively new?
            Standalone VR exists for longer than PCVR did before standalone arrived.
            Minority of headsets?
            Maybe in model numbers, which is why PCVR sucks so much – everyone has a different headset, running through different platform, with different set of buttons.

          • Mike

            Assuming you’re limiting the timeframe to the current revival of VR that started in 2013 with the Oculus DK1, the first standalone was released halfway between 2013 and now – 2018. But the first tethered VR headsets were around 1990.

        • ViRGiN

          Sounds like a super healthy market to me. Hundreds of developers are shitheads with no real skills or vision, or understanding of the market. I know many more hundreds who became millionaires thanks to publishing on Meta.
          Meta plastic toy? Sounds like you worship curved glass Apple Vision which you haven’t even used, and Apple haven’t showcased any killer app usage, other than ‘mind control’ through eye tracking and finger snapping. Video conferences with people who don’t even have Vision? Ridicolous.

          And Deckard rewtards are just that, rewtards.

        • Cl

          People who like gaming will think this headset is useless. I don’t understand how that’s a hard concept.

          MR hasn’t been a thing because the tech wasn’t there yet. You need to realize if it wasn’t for vr gaming and the advances in headsets because of it we might not even be where we are.

          Really you just sound like a hater. Anything you don’t like is stupid and other people that don’t like what you like are stupid, according to you -.-

          Besides, barely anyone tried this yet and you are already talking about how great it is smh

        • Charles U. Farley

          CONTINUE OPERATION: BOOMER DOWNVOTE INTO OBLIVION

        • Arno van Wingerde

          ??? Just because office tools, CAD, etc. are used on PCs, does not mean that gaming PCs are dead.
          Same for VR: I hope we will see a lot of other uses as well, but that does not make gaming VR does not have a future, just that there hopefully will be other stuff as well.

    • Rupert Jung

      I prefer the laserpointer and a reliable controller every day compared with hand tracking. Same reason why I prefer to use a mouse and not moving my hands in front of a webcam. Less reliable, more latency, less precision.

      • Nejham M

        We haven’t tried eye traking as an input element, it sounds like a pretty good implementation though.

    • Charles U. Farley

      Holy shit but you’re a caustic one. How about A. Eat a bag of dicks and B. Don’t forget the ketchup.

    • Ookami

      I use my headset for games. If there are no good games I don’t use my headset. If the my vr headset has archaic laser pointer menus, it doesn’t do anything for convincing me to play or not.

      a steady flow of quality games is absolutely what we need

    • ameba#23234 mdrea

      Quest 3 already has hand tracking navigation. It’s okay but honestly laser pointer is faster and laser eye tracked pointer that clicks when you blink twice would be way faster

    • Killahdillah

      vision pro is garbage. thats why the saudis just invested like half a billion into magic leap right before the apple launch

  • MackRogers

    Imagine how idiotic Apple must be if the rumors of having to pick up Vision Pro in store are founded.

    Imagine the fucking pollyannaish bubble those motherfuckers in Cupertino must live in.

    My Apple Store, for example, is in an area that one might best describe as a fucking warzone. Now I wonder what is going to happen when I pick up my $4,000(after taxes) shoebox size item and walk to my car.

    I really want some woke 25 year old geek squad reject touching my head and face too, talking to me like I am a boomer and reciting the same spiel I could read off the website for onboarding.

    I am not getting it if that is the hoops they make you jump through, I will wait for the aggravated assaults on customers accrue and hits social media and they backtrack 2 weeks later and offer it online.

    • ViRGiN

      I don’t understand what are you so triggered about. You sound like you have issues.
      Also, imagine living in a shitborhood while being able to afford $4000 glass toy.

      • MackRogers

        That’s rich coming from you.

        and did I say I lived there? “My Apple Store” Happens to be a 45+ minute drive away. Which is another ill conceived notion from Tim Apple and the Cupertino liberals, they like to position their stores in city centers like SanFran.

        • Charles U. Farley

          Yeah, cause most regular normal people live in cities. Outside of cities is nothing but cows, MAGAts and bumpkins.

    • Charles U. Farley

      Jesus, you sure sound like a fucking Boomer. Don’t buy it then, we gives zero fucks about Boomer issues.

  • Interesting

  • VrSLuT

    Wish I could remember what I said that just got censored from this forum. That used to only happen with comments about Magic Leap!

  • ameba#23234 mdrea

    Can somebody moderate this comment section or smh? We have a full blown psycho here that ran out of pills and terrorizes the site lmao. Any adults

  • Rupert Jung

    With meta would hide the virtual hands (and controllers) in pass-through mode. It’s highly distracting to see everything twice, out of sync IHMO.