Exclusive: Former Oculus VP of Engineering Demonstrates Long Range VR Tracking System


McCauley says MTS is a ‘vector’ system because control of the laser is done by analog means. The laser isn’t stuck to a quantized grid of possible positions; it can be pointed essentially anywhere within its field of view, only limited by how carefully you can apply voltage (which corresponds to the tilt of the mirror). This means it can be highly precise, even at a great distance.

A look at a tiny MEMS mirror in motion, less than 1mm across.

McCauley showed me the system locking onto a tracking point and maintaining it some 20 feet from the base station. He told me the laser can “easily go 60 feet and track” with the gain turned up (though he did mention it might not have been eye-safe at that amplitude!).

He talks about the strengths of the system compared to a camera-based approach.

“I can’t emphasize enough the computational simplicity and low cost of my system. No frame buffers, large memory arrays, USB 10.0 cables or anything like that,” says McCauley. “Just some control theory, a simple processor and some electronics. It’s also much, much faster than a camera will ever be. It is clearly simpler with much greater range and bandwidth.”

mts jack mccauley vr tracking system laser (7)
McCauley gives a refresher on the principles of Lighthouse.

McCauley is clear to point out that much of what comprises MTS are off the shelf parts and algorithms pioneered for other purposes.

Varjo is Bringing Foveated XR Cloud Streaming to Its Headsets & Beyond

The MEMS mirror is a complex manufacturing task which he says requires a chip foundry to create, though they’re already in production for a range of applications and could be cheap at scale.

“That little mirror can be flicked into position very quickly. It has little mass and it has a powerful actuator. The entire thing is machined onto a single piece of silicon.”

He calls the tracking algorithms, responsible for identifying and following the reflective markers, “well known in the art” and says they were pioneered by professor Kris Pister of UC Berkeley.

“Why nobody [applied this tech] for VR/MOCAP, I do not know. Perhaps nobody thought of it recently but it’s 16 years old and very established. They use these mirrors for optical switches in fiber optic bundles,” McCauley told me. “Sometimes people get stuck in a mode of thinking about a problem. If you learned only to work with cameras then you will use a camera. I’m not a camera guy so I would not consider using a camera so I have to find some other way of doing it.”

There’s more than can be done with MTS beyond what was demonstrated. For one, the laser that’s tracking the object can be modulated for transmitting data to the headset. That could mean that MTS could work (like Lighthouse does now) as a self-contained ‘dumb’ system, which doesn’t need to be connected to a host PC or to a headset. It could simply broadcast the tracking data through the laser, which could be received by anything on the other end, whether it is a VR headset tethered to a PC, or a mobile headset. In this scenario, the MTS basestation doesn’t need any knowledge about the object it is tracking, it just needs to be able to aim well and shoot the right data at it.

HTC Vive Headsets & Accessories Go on Sale for Black Friday

McCauley also supposes the system can achieve full 6DOF tracking from a single marker when combined with data from an IMU in the tracked object. He admits he isn’t 100% on the math for this and says he has someone working on it. My gut tells me this might be technically possible, but would probably lack sufficient correction for IMU drift to be useful. It’s a seemingly moot point anyway, as additional markers are simple to add.

Continue Reading on Page 3…


This article may contain affiliate links. If you click an affiliate link and buy a product we may receive a small commission which helps support the publication. See here for more information.

  • psuedonymous

    This exact same technique was first implemented over a decade ago: http://www.k2.t.u-tokyo.ac.jp/fusion/LaserActiveTracking/

    • mellott124

      Wow, really impressed by this. There are several papers on it at the end of the linked page as well.

    • benz145

      From the article:

      “McCauley is clear to point out that much of what comprises MTS are off the shelf parts and algorithms pioneered for other purposes.”

      “Why nobody [applied this tech] for VR/MOCAP, I do not know. Perhaps nobody thought of it recently but it’s 16 years old and very established,” [McCauley said]

    • Rob B

      Not sure if this is the same group, but just found this with MEMs mirrors for 3d tracking in 2009:


  • Kevin White

    The obvious fix is to have the cameras inside the headset.

    • Hamish Pain

      Except then whatever the cameras on the headset are looking at will get smaller the further away you get, right? And then you have to get the data back to the computer through the headset cables

    • benz145

      We can put cameras on headsets. That’s not the thing preventing VR-capable inside-out tracking.

  • Bryan Ischo

    I fail to see why this is better than lighthouse. The lighthouse units themselves are probably just as cheap to build as this unit, since they have little more than synchronization logic internally and the sweeping laser. The sensors on the Vive headset add cost but they’re just small diodes as far as I am aware and probably not that expensive. Also they can be added to additional devices and track at the same time. How can this single laser track both my head and my hands? Can’t do it.

    I like that people are looking for better solutions but … just make lighthouse cheaper via volume production and refinement. There’s your tracking solution, now go focus on something else. Like, finding a way to eliminate the display cable. Or improving the field of view. Or improving the resolution. Those things are desperately needed. Better tracking than lighthouse, not so much.

    • Tyler Cook

      If this works like the sony laser pico projector, then it could track multiple objects. The way it works is like a CRT screen. It paints an image by sweeping a laser using the MEMS in a rectangle pattern. If there were two points to track, it would get both as it sweeped.

      It is very similar to lighthouse, except it is sweeping in a more precisely controlled area.

      But, I do agree with you in that it doesn’t really need to be that precise. Lighthouse floods the area, which is exactly what you want anyway, since you can be anywhere inside the zone.

      The only big benefit I would see is if, for some reason, sweeping a laser in this fashion is somehow more efficient or allows for greater distance in tracking. Then, you can have bigger rooms.

      • Bryan Ischo

        Good points; however, you can have bigger rooms with more lighthouse units and some synchronization logic and/or software to stitch the swept areas together.

        Also, I’m pretty sure that you could track 500 sensors using lighthouse if you had enough sensors. You can just keep adding sensors indefinitely, they can all share the same tracking signal from the lighthouse units. I doubt that the solution described in this article could go beyond a few tracked points, even if it used the sony laser pico projector technique you described.

        Also there’s the fact that the technique described in this article requires a reflection of laser light back to the unit; there is room for lots of error there.

        • Jack McCauley

          I have two systems, a vector based one like the film, and a raster scanned one like the pico projector. Either one will work about as well as the other.

          • Bryan Ischo

            Thank you for replying. My goal is in no way to discourage your work, if you can make better/cheaper tracking, by all means do!

          • Eric B

            Hi Jack, are you still working on this tracking ? I think it’s fantastic and it would lead to mass adoption of Virtual reality .

      • Jack McCauley

        Exactly. I’ve got one of those to experiment with, an oscillating mirror.

      • Sven Viking

        As presented here, though, each complete sweep is taking seconds rather than milliseconds.

    • David Mulder

      The big advantage seems to me to be that the cost-per-tracked-item will be FAR lower. So if you would want to do full body tracking then this is a solution that might do the trick if it’s fast enough, whilst Lighthouse is simply impossible and Camera based tracking unlikely. The disadvantage of this system is that it has – like Oculus’ camera tracking – a far bigger occlusion problem, so you might end up having to put a fair number of these stations in your room for reliable tracking of more than a headset.

    • Jack McCauley
      • Rob B

        Have you considered placing one mirror on the tracked object (headset), and searching for a fixed lit up marker? That way multiple objects may simultaneously track themselves without interference. (ie, they all try to point to the same bright fixed object).

    • Jack McCauley

      A couple of things, the laser energy is spread over an ellipse and with mine it’s a point source so the range is much longer. What do you think four motors cost??

      • Ryan

        I’m actually very surprised Vive came in as cheap as it did, given the cost of motors and lasers. Micro fabricated mirrors should be cheaper, but do they have the tip/tilt range to track objects close to the mirror?

        • Jack H

          I haven’t seen MEMS raster mirrors with FoV better than about 14 deg.

      • Hamish Pain

        @BryanIscho:disqus Advantages as far as I can see them:
        Longer range (Lighthouse needs an LED flash), automatic visual acknowledgement of tracking (a blessing for devs like me, though that will probably change to an IR laser, right?), far higher update frequency (lighthouse needs three phases of light emission, so there can be some drift in the meantime+need for IMU).
        Harder to track multiple regions (maybe? If the mirror’s steerability has a great response time and low overshoot [MEMS!], then it could probably be adapted to continuous scanning + tracking multiple items at reduced update frequency), still needs to be hooked up to a computer or the tracked object needs a light-sensor for information transmission, reliability of MEMS steering vs. motors (MEMS may be better due to low-mass and no friction issues), doesn’t track orientation currently or depth (though I can think of a few ways to do so with what I’m assuming to be the actual tech behind it. FPGA, yeah?)

        @jack_mccauley:disqus Love this kind of high-speed tracking. Have you considered using retroreflective materials for the marker? That could increase range even further by reducing laser-spread on impact. It could also allow a second laser-tracker unit to target each marker by reducing light-interference without having to alternately-pulse the lasers. I’m assuming the visual sensor, if not a camera, is an opto-diode under a lens for this, as spatially disparate laser light would still interfere with it.

        Pretty awesome laser steering, MEMS has really sped things up. I remember making a spinning mirror based laser tracer, what I wouldn’t have given for a steerable mirror!

        • benz145

          To answer a small part of this, some of the tracked objects in the video were retroreflective markers. Jack showed me a few different things being tracked.

    • Andrew Jakobs

      One reason this is better, it has no actual physical moving parts like the motors for spinning the lens on the lighthouse (which makes noise and will wear out after a while, at this point we have no idea how long the lighthouse basestations will work)..
      And the mirror is just like a DLP chip, so multiple mirrors can track multiple object, BUT you’re right, in that regard the lighthouse system is much simpler as you just add sensors on the stuf you want tracked and don’t need extra lasers for that..

    • Chip Weinberger

      Another limitation to lighthouse is the size of the objects it can track.

      Look at the size of the ‘halo’ on the vive controllers. Thats about the limit for good tracking.

      This coupd possibly track smaller items. It is also cheaper to make a tracked object. You could put these reflective markers on lots of objects. Without the need for electronics in them. Put a marker on whatever you want and have high fidelity tracking on it.

  • VR Geek

    Sounds like there may have been a few egos battling over which way to do the tracking at Oculus. Not sure Oculus went the right way with their camera based tracking. We will see in the final product, but it will need to be WAY better than the DK2 which would require IMO something much greater than the 752×480 camera they shipped with it. Based on this article and my own personal experience with the DK2, I suspect even 1080×1920 would not give enough resolution to track when back 10 feet. Maybe 4k plus would, but then there are larger bandwidth demands and more painfully, massive computational efforts required by the host PC. My gut is telling me that once the dust settles in May or June, how big of an issue, or not, the constellation tracking system is will become very apparent. I sense Oculus is in trouble here, especially after using the ViVe. I hope not as they really have done sooooo much to get VR off the ground.

    • DJ

      Sometime after the DK2 was released, Palmer Luckey explained that they went with the camera tracking system (now known as Constellation) after they’ve tried everything that was viable at the time. Apparently they didn’t try just a few, they tried dozens of technologies. Many tracking systems are better than Constellation, but they’re either too expensive, or locked by patents and unattainable, or have problems that only become apparent in application. The one technology that matched all of their criteria for price, availability, and capability better than any other that they tried was Constellation. It was a basic engineering decision.

      It’s easy to watch at all of these impressive tech demos and say, “That’s the solution to everything!” But, they rarely actually are solutions to your very specific application.

      I think that Constellation probably won’t survive more than a couple iterations of the Oculus Rift. Better systems are coming down the pipeline, you’d be ignorant to think that Oculus isn’t working with them to determine their viability. And new ideas are being invented that might even supplant those in a few years more. It’s a very volatile, and exciting, area of research at the moment.

      • VR Geek

        I very much agree with all you comments. That said, Valve is about to own Oculus pretty hard with Lighthouse. You cannot blame Oculus as they surely tried their hardest. It is just interesting that even with tons of money, and top talent that they are coming to market with what looks like the inferior tracking system. I am sure someone over there is loosing sleep. I hope constellation is better than previous demos when the CV1 arrives and that they quickly address if not for CV2.

        • Rob B

          From my understanding, the resolution of the camera isn’t as limiting as you describe. Its still used in conjunction with the IMU unit, and only used to correct larger scale drift.

          • VR Geek

            I can only speak to the DK2 camera myself which also uses the IMU, but it has never been that solid. We will have to wait to try the CV1. Lighthouse was incredible when I tried it extensively last year. Super solid.

          • Rob B

            Here’s the comments from OKreylos (doc-ok), that describes the process if you’re interested:


          • VR Geek

            Interesting. Thanks

          • Guest A

            It’s true that the camera isn’t as limiting as it sounds, but of the two (camera and IMU) the camera is the limiting factor. And like you said, they have to be used in conjunction. IMU is very fast but not accurate. The accuracy degrades overtime (i.e., drifting). How many seconds before it no longer tolerable as 1-1 depends on the specifics of the IMU. You need the camera to give you that accuracy. The higher the resolution the camera is, the further the range it can give you that accuracy data. However this means if you want longer range you need higher resolution and this doesn’t scale well.

  • Foreign Devil

    I keep thinking about the military applications of a laser that can lock in perfectly on a rapidly moving target. . .

  • Po Tato

    Celebrities gonna freak out when their fans track their each and every move with this kind of device

  • TrevorAGreen

    What I’m curious about is hybrid tracking systems that are domain specific. So If I have a lighthouse system I can track the vive and the controllers. And any other lighthouse enabled device. But what if I want to bring in something else that is tracked in the same space? Say I have a coffee mug. And I want to see that. That might be better tracked by a camera than enabling it for lighthouse. Or maybe I buy a item that is specifically designed to match a certain tracking approach. So it has some sensor and the object included, but it still allows it to appear in the same 3d space. Maybe that is a lower resolution of tracking. Maybe something even higher. Something specific that would be a cool tactile experience would be foam balls. The games that you could apply that too would be almost endless. But they probably wouldn’t be appropriate for a hard body powered solution like lighthouse. They would need to be camera tracked, or some other system. We are getting the vive and the controllers. Now we need to ramp up and start creating other tactile experiences.

  • Fadelis01

    The main issue I would see with this system is the fact that you need to track and differentiate multiple objects in the play space. I do see the need to move as much of the processing burden from the equipment in the players hands/head. That is the one benefit that truly remains with the CV1. I have high hopes for this technology, and this guy is on the right track IMHO.

    • Fadelis01

      Hey! Brainwave moment… what if the mirrors and lazer detectors took advantage of polarization? This would allow your scanners to scan across a mirror with a custom “polarization pattern” and read orientation and even an object identification key of some sort. I’m imagining a mirror “strip” that would have multiple polarization regions. As the mirror was scanned, a serial string of information could be created in the reflection as the Lazer reflection passed across the mirror. This could verify that the mirror is the “right” reflective surface for that scanner as well.

      • Fadelis01

        A mirror “strip” could have multiple states at that point and not just binary the polarization tint lines could look something like this for example /|/ where the two leading and trailing “/” polarization angles would denote the beginning and end of a string. And the middle angles “|” would be some defined but of info about the object the mirror is attached to. The scanner could trial multiple angles of sweep until it reads a full strip. At that point, it could infer object type, angle, position and distance from the scanner.

        • Fadelis01

          You could even have one contiguous mirror “halo” with a polarization “Descriptor phrase” going all the way around. At that point, the scanner simply finds the reflection, then finds the angle where it can “read” and the phrase that is visible could also state the rotation markers for the object… The mirror could utilize the same “microsphere” technology used in reflective safety paint and the Polarization could be a simple plastic polarizing strip that would just “glue” on top of that painted finish. This would be very cheap and robust.

          • Fadelis01

            The paint idea brings up other thoughts… what is there were phosphorescent “dots” that could be charged with a low level UV stobe. An optical camera could then broadly view the space and direct a more precise lazer to the right points of interest. This would still require no power or logic from the tracked objects, but would further protect the users from getting blinded by lazer scanning.

        • Fadelis01

          AND if the polarization phrase was done via a transparent LCD overlay, then button/trigger info could be relayed via the same positional scan back to the room scanners…

  • MosBen

    What is the value in a greater range? With the Vive at room scale everyone’s concerned about not having enough space for VR. What does a greater range allow us to do that I’m missing?

  • OgreTactics

    How precise is it? I mean: how small can be the reflective marker and still be tracked by this system?

    Because if precise/small, then you could very well have a 3 laser array projected by the mirror onto a surface with 3 tiny markers side-by-side for xyz movement/orientation tracking. But of course you ultimately would have to use different marker areas and MTS to track the whole 360° movements of an object.

    Anyway, I’m convinced this can be WAY smaller and cheaper than lighthouse. In fact I don’t down to how small it can be reduced (until it can be integrated into the headset) which lidars will never be.

  • Eric B

    i wish this tracking system was available .Any chance Jack is still working on this?

  • Nice post, thanks for sharing.
    News Health Education Entertainment Technology Writing