Augmented reality, while growing in buzz, has been around in some form or another for more than two decades. But immersive head-mounted AR for consumers has had one major hurdle that Microsoft may have just solved.
Despite copious concept videos showing a far flung future of AR—and way more smartphone/fiducial marker-based novelties than I care to count—augmented reality’s big hurdle is a self-contained ‘inside-out’ tracking system (one that doesn’t rely on external sensors) that could provide tracking that’s fast and accurate enough to allow augmented objects to appear anchored in the real world such that they actually seem real. When accuracy and latency isn’t low enough, the augmented objects swing around with your head and your brain doesn’t consider them part of the world.
For a long time it didn’t seem like real progress was being made on inside-out tracking even while it was being made in other important areas of AR like display technology. Inside-out tracking has remained for years an elusive key to the success of AR, but it seems like Microsoft has made a game-changing stride with HoloLens.
It seems many have been too distracted by the headset’s all too narrow field of view to realize how jaw-droppingly good the device’s inside-out tracking is. It was so good compared to anything else that I’ve seen that I actually waited until a second time to get my hands on HoloLens to double-check what I saw initially. On that second check, it was confirmed: rock solid inside-out tracking. The kind that’s so good that you can shake your head back and forth much faster than you’d normally have reason to, and see that the augmented objects in the environment appear locked in place; the same kind of tracking quality you’d experience with top outside-in tracking systems like the HTC Vive and Oculus Rift. HoloLens’ inside-out tracking is better than anything we’ve seen from other big names in AR, including Magic Leap and Meta.
The video above gives a good idea (but actually looks slightly less impressive than what it looks like through the headset) of how solid the inside-out tracking is; the virtual screens stay nearly locked in place as the user moves their head.
The question of robustness (the ability of the tracking to work consistently in a wide range of environments and lightning conditions) still remains open. I’ve only had a chance to test HoloLens in a select number of environments, but one of those places was a veritable torture-test for any inside-out positional tracking system: on a busy conference show floor.
It was several months ago, my first time getting the device on my head, in the middle of the show floor with people buzzing about, walking directly passed me. I popped HoloLens on my head and placed a few floating windows around the area.
Normally I would expect all the moving people to seriously throw the system off; it has to look at certain parts of the environment and somehow tell what’s moving and what’s static, even though to a camera the whole world appears to move when you move your head. This is a fundamentally challenging computer vision problem. But in my time with HoloLens, it easily ignored the people walking all around me, and kept the windows and other AR objects stuck convincingly where they were supposed to be. So things are looking good for robustness.
Another reason there hasn’t been much buzz about HoloLens’ game-changing inside-out tracking may be because it wasn’t always this good, crucially back when Microsoft first introduced the device to an initial swath of press back in early 2015. According to a source working with the headset, the tracking has indeed improved with time.
While the tracking on HoloLens is outstanding, the device falls short of being a total breakthrough in AR due to the diminutive field of view and clunky gesture input. A next-generation HoloLens which fixes those criticisms will blow the lid off of AR as it stands today.
AR is isn’t the only thing to get excited about when it comes to inside-out positional tracking being ‘solved’; virtual reality too would massively benefit, opening the door to untethered virtual reality headsets with huge (potentially limitless) tracking volumes. It’s what mobile VR headsets like Gear VR (which lacks positional tracking) have been eagerly waiting for.
If Microsoft’s inside-out tracking success can be replicated, the next step is making it affordable. With the HoloLens development kit currently priced at $3,000, there’s a way to go, but the critical problem of inside-out tracking may finally have a solid existence proof.