A new photo of Magic Leap 2 appears to show the device’s controller equipped with cameras for inside-out tracking which would be the first time we’ve seen the approach employed in a commercial XR headset.

Though we learned plenty of interesting details about the forthcoming Magic Leap 2 AR headset back in January, it looks like there’s still some secrets left to uncover.

Image courtesy Peter H. Diamandis

A recent photo of Magic Leap 2 posted by Peter H. Diamandis is, as far as we know, the first time we’ve gotten a clear look at the front of the Magic Leap 2 controller. The photo clearly shows what appear to be two camera sensors on the controller, indicating a high likelihood it will have on-board inside-out tracking.

Image courtesy Peter H. Diamandis

The original Magic Leap 1 also had a motion controller, but it used magnetic tracking. This was the reason for the curious square sticking out of the headset’s right side (it contained a receiver that sensed the magnetic field emitted from the controller).

A square dangling off the side of Magic Leap 1 senses a magnetic field created by the controller | Image courtesy Magic Leap

Magic Leap 2 has ditched the square and appears to be moving to entirely on-board inside-out tracking for its controller. To our knowledge this will be the first time a commercial XR headset makes use of the approach; that is, assuming Magic Leap 2 beats Meta’s Project Cambria to market (the latter is also expected to use on-board inside-out tracking for its controllers, based on some leaked details, though it hasn’t been confirmed yet).

Other standalone XR headsets, like Quest 2, typically use headset-based inside-out tracking to track their controllers (or the user’s hands). That is: cameras on the headset look for the controllers (typically arrayed with IR LEDs) and use their location to map their movement relative to the headset.

Image courtesy Oculus

While this approach has proven effective, it only works when the controllers are in view of the headset’s cameras. That means it’s possible for the headset to lose track of the controllers if they spend too much time outside of the camera’s view (like if you held them too low, too high, or behind your back). Putting cameras on the controllers themselves would enable inside-out tracking that, in theory, has coverage no matter where the user holds it.

Image courtesy Oculus

Beyond ‘anywhere’ tracking coverage, putting inside-out tracking directly on the controller means the headset wouldn’t need to be covered in so many cameras (Quest 2 uses four cameras while Rift S tops that at five), and it means the elimination of the ‘tracking rings’ (seen on most standalone headset controllers) making them sleeker and perhaps less prone to breakage.

There’s more potential benefits too. Giving controllers their own inside-out tracking could make it easier to use a combination of input methods, like a controller in one hand and controllerless hand-tracking in the other. Currently that’s a challenge because the cameras on the headsets tend to use different exposure settings for tracking controllers compared to tracking hands. Furthermore, decoupling controller tracking from the headset has the potential to reduce controller tracking latency.

But, as they say, there is no free lunch. Giving controllers on-board inside out tracking likely means putting a dedicated processor inside with enough power to crunch the incoming images and compute a position, which we’d expect to result in higher costs and more battery drain compared to having simple IR LEDs on-board.

There’s also the issue of motion blur for the on-board cameras. While inside-out tracking has been shown to work well for headsets, controllers can move considerably faster, especially during intense games—so fast, in fact, that it prompted Valve to update its SteamVR Tracking tech to account for the speed of top Beat Saber players.

As you can imagine, swinging a camera around that fast is likely to induce motion blur across the image sensor, especially in darker scenes where the exposure time needs to be increased to gather more light.

SEE ALSO
'Resident Evil 4' for Quest 2 Gets 'The Mercenaries' DLC Today for Free

Of course, Magic Leap 2 isn’t intended for gamers, so it’s possible that this wouldn’t be an issue for the headset’s enterprise-focused use-cases. Or maybe the image sensors on the controllers are fast enough to avoid motion blur even during quick movements?

Perhaps most likely the sensors aren’t above average, but the controller will simply lean primarily on IMU-based tracking until the controller slows down enough to get a proper position (this is the same approach that existing standalone headsets use to deal with controllers that occasionally leave the headset’s field-of-view).

In any case, we’ll know more as Magic Leap 2 approaches commercial availability. The headset is expected to launch this year but a specific release date hasn’t been announced yet. In the tweet which included the controller photo, Peter H. Diamandis said he would be hosting Magic Leap CEO Peggy Johnson at his Abundance 360 event at the end of April which could signal the next time the company will divulge official details on the headset.


Editor’s note: Before someone in the comments says ‘SteamVR Tracking tracking controllers have been doing inside-out tracking for years!’ the industry vernacular does not consider systems using artificial markers as ‘inside-out tracking’.

This article may contain affiliate links. If you click an affiliate link and buy a product we may receive a small commission which helps support the publication. See here for more information.


  • Lulu Vi Britannia

    Very interesting take on the self-tracking controllers ! The motion blur could be a problem indeed. I guess we’ll have to put it to the test in Beat Saber Expert+ songs, once it’s available. I still think it will be an improvement over Quest 2 and similar inside-out controllers, but it will definitely drain faster.

  • Cless

    IF it wasn’t a battery/price killer, that would be the next step for controllers for this kind of tracking.

  • Nepenthe

    I’ve been saying for years we need inside-out tracking for the controllers.

    “Other standalone XR headsets, like Quest 2, typically use headset-based inside-out tracking to track their controllers” — wouldn’t ‘headset-based inside-out’ otherwise be known as ‘outside-in’ tracking? I mean, if the cameras doing the tracking are on something outside the object being tracked…

    The technology / cameras will need very rapid updates to avoid motion blur, so who knows if it’ll be feasible for all applications any time soon, but I think it’s where it needs to go eventually.

    • Cambria has inside-out tracked cams in them.

    • Bob

      Meta’s next headset, which is slated to launch sometime this year, has “inside-out” tracked controllers. They use cameras on the controllers themselves to deduce its position.

    • benz145

      Yes if you use the headset as the frame of reference and assume only the controllers are moving then it would be conceivable to call it outside-in tracking from the controller’s perspective. However, in the case of a headset like Quest, we don’t know the position of the headset without inside-out tracking, so ultimately the absolute position of the controllers can’t be known without relying on the headset’s inside-out tracking.

      • VR5

        The position of the controllers relative to the headset is tracked outside-in and then added to the absolute position of the headset, which is tracked inside-out. So it uses a combination of inside-out and outside-in tracking for the controllers.

  • VR5

    Editor’s note: Before someone in the comments says ‘SteamVR Tracking tracking controllers have been doing inside-out tracking for years!’ the industry vernacular does not consider systems using artificial markers as ‘inside-out tracking’.

    Maybe that’s a problem with industry vernacular though? It’s inconsistent, the term outside-in tracking is used despite that universally requiring “artificial markers” (light bulbs on PS Move controllers, LEDs on Oculus Touch controllers, light markers on PSVR headset, LEDs on Rift headset).
    Many inside-out headsets use markerless visual inside-out tracking that can process random geometry. Lighthouse uses timed signals that triangulate position from the distance/time it took to reach the inside-out sensors.
    Position is either tracked inside-out or outside-in, it has to be one or the other.

    As for motion blur impacting tracking, I thought it’s the motion sensors (accelerometers and gyroscopes) that do the heavy lifting and the position tracking is only confirming the position at a lower poll rate (with inbetween changes estimated by sensors with high accuracy)? Also, if the cameras are low enough latency shouldn’t blur be avoidable? Obviously further raises cost.

    • But what ‘signals’ is the lighthouse system trying to triangulate? Signals from specialized hardware outside the playspace facing in. It doesn’t necessarily matter who’s doing the ‘looking’. The inherent question these terms are trying to answer is, “Do I need specialized external hardware in order for tracking to work on this headset?” In that sense Lighthouse is still outside-in.

      • VR5

        It does matter who is doing the tracking. The lighthouse base stations don’t read any data so they cannot track anything. They shower the room with reference data for the controllers to be able to track their position from the inside-out.

        When you want to say, “it doesn’t require peripheral devices” you could just say that. Misunderstanding that this is implied by inside-out doesn’t change the facts of the technology. There are several better suited terms like the recent self-tracking that you could use.

        • Calling it self-tracking doesn’t solve the issue by your own standards. You just argued that Lighthouse controllers track their own position too. Which I think shows you’re not actually trying to find a rational solution. You just want to distance the conversation from inside-out vs outside-in.

          Here’s my take on what’s actually going on here. You bristle at Lighthouse being lumped in with other outside-in tracking solutions that have all fallen by the wayside and you worry Lighthouse is next, and with it the high quality tracking you’re accustomed to. The semantic arguments are your copium.

          • VR5

            You aren’t wrong and it occured to me as well after posting it, even before someone pointed it out (thanks anyway for being that somebody). I guess if you take self-tracking as an abbreviation of self-sufficient tracking, it would do the job.

            Anyway, visual inside-out tracking isn’t completely self sufficient either, it needs a light source for the environmental data to be visible, and although the sun is hardly peripheral hardware, the lamps you have installed in your house are. They are similar to lighthouse base stations, enabling the tracking of your environment, although lighthouse provides referential data that is easier to compute, so it does more work than a simple light source. But it needs to be set up as opposed to lamps which are usually already available and ready.

            The hassle of having to set up extra hardware and making room, including it in your room decoration, is what makes people want a solution like visual inside-out tracking.

            Linguistically speaking, there is no problem with laymen misuing terms if it serves to communicate what they want or appreciate. But technically speaking saying lighthouse is outside-in, or saying Oculus Touch controllers are tracked inside out, is simply wrong. And I will never encourage that use or pretend that it is okay. I might not (actually very likely will not) mention it or start an argument because fighting an army of ignorant people is just useless and a waste of energy.

            But I was in a mood to argue this and I think when speaking to people who understand the technical background it isn’t as much of a waste of energy than when talking to a layman who just doesn’t want to admit they are wrong. Although for a person who should know better there is also more at stake. Ben hasn’t said anything wrong as far as I’ve seen though.

    • Cl

      If its needs an external device its outside in to make it easier to understand. Dont complicte things. You know exactly what people mean when outside in is said. Would you rather things be labeled “inside out tracking with external lasers” and “inside out tracking with internal cameras” and ” outside on tracking with external cameras” and ” outside in tracking with magnetic fields”. Its confusing and complicated. If you simply say steamvr is inside out with no other explanation people will be confused with the external sensors and say its false marketing. Just accept it.

      • VR5

        You’re right in so far that to avoid confusion it is best to avoid speaking of technical details that will not be easily understood. But technical jargon will always be more correct than colloquialism based on misuse. So at least avoid that.

    • Nepenthe

      Maybe that’s a problem with industry vernacular though?

      My take is said vernacular is dumbed down. Always seemed to me a VR enthusiast site shouldn’t be dumbed down. Oh well.

      • VR5

        I think it’s the vanity of the ignorant. The ignorant will rather insist that their wrong assumption is right than ever admit to fault. And websites have a lot of these vain ignorant readers.

        But as we’re seeing with the climate catastrophe, while the truth can be ignored it won’t go away. It doesn’t matter if you understand the technical facts, they will forever be facts.

    • benz145

      It is one or the other; the industry considers SteamVR Tracking to be outside-in.

      You’re right about the IMUs doing most of the heavy lifting. In most outside-in systems they’re getting a ground-truth correction from an absolute positioning method (ie: SteamVR base stations), usually in the realm of 50-100Hz.

      Let’s just suppose that controllers with on-board inside-out tracking also update at 100Hz (every 10ms). If you swing your arm fast while doing something like playing Beat Saber, and we assume this blurs the cameras, you could easily have gaps of several seconds (thousands of milliseconds) where the cameras can’t get establish their position for a ground-truth correction.

      This is probably already an issue on headsets like Quest when the controllers are moving fast and are potentially blurry as seen through the headset’s cameras. However moving the camera’s vantage point onto the controller itself could exacerbate the issue greatly because the rotations are so much faster from the hands compared to a controller passing by the headset’s cameras.

      • VR5

        Lighthouse is not outside-in. And I’ve seen Jamie on UploadVR argue that lighthouse is neither.

        It is true that when asked about adopting inside-out tracking, Valve spokespeople didn’t argue that they already did. I can only guess why but probably because they didn’t want to create the impression that they were using the same tech. Which they aren’t, it is a different implementation of inside-out tracking.

        If you want to explain these things and be both technically correct and avoid being confusing it is best to completely avoid this terminology and refer to the most relevant aspects of the tech.

        But I guess Magic Leap wants bragging rights for being first to something, otherwise you would probably have used the term self-tracking as with the Cambria controllers.

        All you need to add is “visual”: first controllers with “visual” inside-out tracking. But of course then you’d have to explain that because many readers would be lost why “visual” deserves mentioning.

  • Ad
  • Anonmon

    The other problem with a “Cameras on the controllers” approach that the article seemed to miss was, you introduce independent inside-out tracking to separate devices with no way for each device to track each other, and you result in every device existing in its own tracking universe. Which anyone who tries to mix Lighthouse stuff with Facebook stuff can attest to being a pain in the butt with drifting.

    There’s zero way around it, to have a good time the controllers NEED to know where the HMD is, and vice versa. Not a difficult problem to solve, all inside-out HMD’s have been doing half of that equation since their inception, but it’s not something to gloss over. The other way to solve that is to scan your room, save that somewhere (Not exactly great for privacy. You know, the exact reason among many why Facebook hardware is just, no.) and have each device reference that to get a decent lock as to where they are in the scanned room. Loads of bandwidth to pass that around, loads of processing on top of the processing already for inside-out, all around a lot of compute for not a lot.

    It’s the same reason why “Well just slap cameras onto trackers” for full body and such can’t work. They’ll have excellent tracking by themselves sure, but they’ll still drift around from each other, no matter how good the tracking is.

    • brandon9271

      What if you put a few IR leds on the controllers that the HMD could see? Then it could sync everything together

      • benz145

        Definitely viable to have those as a fallback, though as we’ve seen with other controllers this usually necessitates a bulky design to get the LEDs somewhere that are easy for the headset to see.

        • brandon9271

          I was think of it as more of a way to calibrate the individual coordinate systems Anonmon was talking about. It would be a way for the individual devices to not only know position in space but also in relation to each other. It should only need to be done once.. but who knows maybe that’s not even be necessary.

      • VR5

        Apparently the XR chip can link with other XR chips to share detected geometry and get absolute positions for all devices joined in a tracking network in the same environment. So that isn’t necessary anymore.

        • brandon9271

          oh wow! That’s awesome! Would be cool to have games with multiple people in the same room as well.

    • dk

      Cambria will have this too …most likely has a few ir light too and from time to time the headset is tracking the controller

    • benz145

      You’re right that does sound like an interesting challenge to solve.

  • Thud

    They could conceivably track the whole legs using only 2 trackers mounted to the calf using this concept rather than 4 (2 for feet and 2 for knees) as is needed now.

  • dk

    it would be interesting to know if it works with 2 controllers unlike ml1 which had just one

    • benz145

      Presumably it should be easier to achieve than ML2 given that they’re moving away from magnetic tracking. Their prior ‘reverse’ magnetic tracking approach I expect would be harder to use with two controllers because it would require two transmitters instead of one transmitter and two receivers (as we’ve seen with other magnetic tracking).

  • Actually it was possible to see they have inside-out tracking in a presentation that Magic Leap gave two weeks ago. I added the news of inside-out tracking in my weekly roundup of my blog on Feb, 28th. You don’t read my roundups, Road To VR journalists… this makes me sad :(

    • silvaring

      Just ask Ben to retweet them when you post them, I’m sure he would be happy to do so.

    • benz145

      Good spot Tony, I definitely missed the clip showing the controller in detail!

  • mirak

    Editor’s note: Before someone in the comments says ‘SteamVR Tracking
    tracking controllers have been doing inside-out tracking for years!’ the
    industry vernacular does not consider systems using artificial markers
    as ‘inside-out tracking

    If you go down this path, then the Nintendo Wiimote was doing it 16 years ago, way before the Steam VR Tracking.

    The wiimote really has an embeded camera to do the tracking.
    But yeah it tracked infrared pictures of the leds of the sensor bar, not visible light images.

  • Wow, hand controllers. Our little AR blip is finally growing up.

    Hand controllers should be the ABSOLUTE BARE MINIMUM for anything VR or AR related. Hand tracking is nice, but not a substitute.

    And not having either is just unforgivable.