Researchers Show Full-body VR Tracking with Controller-mounted Cameras


Researchers from Carnegie Mellon University have demonstrated a practical system for full-body tracking in VR using cameras mounted on the controllers to get a better view of the user’s body.

Although it’s possible to achieve full-body tracking in VR today, it requires the use of extra hardware that needs to be strapped onto your body (for instance, Vive Trackers or IMU trackers). That makes full-body tracking a non-starter for all but hardcore VR enthusiasts who are willing to spent the money and time to strap on extra hardware.

Three vive trackers are used here to add torso and foot-tracking | Image courtesy IKinema

Because standalone VR headsets already have cameras on them to track their position in the world and the user’s controllers, in theory it’s also possible to look at the user’s body and use a computer-vision approach to tracking it. Unfortunately the angle of the cameras from the headset is too extreme to get a reliable view of the user’s legs, which is what led Meta to recently conclude that full-body tracking just isn’t viable on a standalone headset (especially as they get smaller).

But researchers from Carnegie Mellon University are challenging that notion with a prototype standalone VR system that adds cameras to the controllers to get a much clearer view of the user’s body, making it possible to extract reliable tracking data for the legs and torso.

What’s especially interesting about this approach is that it seems to align with the direction next-gen VR controllers are already heading; both Meta’s Project Cambria and Magic Leap 2 are using controllers that ditch a headset-dependent tracking system in favor of calculating their position with their own inside-out tracking system.

Image courtesy Carnegie Mellon University

Using a standard Quest 2 headset as the basis for their prototype system, the researchers added two cameras to the controller which face the user. With the user’s hands in front of them, the cameras can get a much clearer view of the upper and lower body. This view is corrected so a computer-vision system can optimally extract the user’s pose and then combine that data with the known position of the head and hands to create a full-body tracking model.

Image courtesy Carnegie Mellon University

Of course, the user’s hands won’t always been in front of them. The researchers say some limited testing showed that VR users have their hands out in front of them around 68% of the time. When the hands aren’t in a good position to capture the body, the system should fall back to an IK estimate of the body position. And though their prototype didn’t go this far, the researchers say they believe that with an additional camera angle on the controller, it should be possible to capture the leg position even when the user’s arms and controllers are resting at their side.

As for accuracy, the researchers, Karan Ahuja, Vivian Shen, Cathy Mengying Fang, Nathan Riopelle, Andy Kong, and Chris Harrison, say that millimeter tracking precision is probably out of the question for this sort of system, but centimeter tracking precision is likely on the table, which may be good enough for many VR use-cases. For their prototype specifically, the system had a “mean 3D join error of 6.98cm,” though the researchers say this should be “considered the floor of performance, not the ceiling,” given the limited time they spent optimizing the system.

With full-body tracking, legs are finally a viable part of the experience. That’s desirable not just to make your avatar look more realistic to other people, but also the option to incorporate your lower body into the experience, adding to immersion and providing another input for players to use in gameplay.

Image courtesy Carnegie Mellon University

The researchers not only created a full-blown tracking model for the system, they also made some prototype experiences to show off how tracked legs can add to gameplay. They showed a hockey goalie experience, where players can block the puck with any part of their body; a ‘body shape matching’ experience, where players match the shape of an incoming wall to fit through it; and even a ‘Feet Saber’ game, where players cut blocks with their hands and feet.

– – — – –

So could we see full-body tracking from headsets like Magic Leap 2 and Project Cambria? It’s tough to say at this point; although the controllers appear to do their own inside-out tracking, the cameras on the controllers seem to point mostly away from the user.

But maybe some future headset—or just an upgraded controller—could make it happen.

Regardless of where those headsets land, this research shows that practical, low-friction full-body tracking on standalone VR headsets might not be that far out of reach. Combined with the ability to run highly realistic face-tracking, the standalone headsets of the future will radically increase the embodiment felt in VR.

This article may contain affiliate links. If you click an affiliate link and buy a product we may receive a small commission which helps support the publication. See here for more information.

Ben is the world's most senior professional analyst solely dedicated to the XR industry, having founded Road to VR in 2011—a year before the Oculus Kickstarter sparked a resurgence that led to the modern XR landscape. He has authored more than 3,000 articles chronicling the evolution of the XR industry over more than a decade. With that unique perspective, Ben has been consistently recognized as one of the most influential voices in XR, giving keynotes and joining panel and podcast discussions at key industry events. He is a self-described "journalist and analyst, not evangelist."
  • Considering that fullbody tracking was teased for Cambria, I want to hope (but I think not) that is something like this

    • Patrica R. Rugg

      I’m acquiring 108 US dollars/hr to finish little tasks on-net.~ol80~I not the slightest bit accepted like it tends to be feasible anyway my dear companion gathered $27k in no less than about a month and a half effectively finishing this simple proposition besides she has persuaded me to join.~ol80~Check new subtleties on following site.

      – – – – >>>> https://0se­.­co/ntzqQ

    • lujho

      The Cambria controllers have cameras. It probably won’t have this body tracking at launch, but seeing what they did with hand tracking on the Quest, it’s not out of the question that they could introduce it with a software update at some point. The have added so many features to the Quest since launch, I expect that to continue with their other hardware.

  • Andrew Jakobs

    Hmmm, I have my hands a lot next to my body as it’s a big strain to hold them up for a long time. It’s an interesting method, but petrsonally I think a wireless kinect would be much better and is not that much of a hassle to have on spot somewhere in front of you.

    • Jeremiah Tothenations

      Agreed, most people would be OK with one camera on a desk I reckon.

  • I really like the concept. If we want a setup that doesn’t take preparation time, it’s great.

    But still, in practice, I would prefer an optional external camera that could be used for increased tracking reliability (I often loose tracking when I’m too close to uniform color walls), full body tracking, and mixed reality recording.

  • Jake

    Already exists!

    WMR + Kinect + K2VR