Augmented reality, while growing in buzz, has been around in some form or another for more than two decades. But immersive head-mounted AR for consumers has had one major hurdle that Microsoft may have just solved.

Despite copious concept videos showing a far flung future of AR—and way more smartphone/fiducial marker-based novelties than I care to count—augmented reality’s big hurdle is a self-contained ‘inside-out’ tracking system (one that doesn’t rely on external sensors) that could provide tracking that’s fast and accurate enough to allow augmented objects to appear anchored in the real world such that they actually seem real. When accuracy and latency isn’t low enough, the augmented objects swing around with your head and your brain doesn’t consider them part of the world.

mts jack mccauley vr tracking system laser (9)
See Also: Former Oculus VP of Engineering Demonstrates Long Range VR Tracking System

For a long time it didn’t seem like real progress was being made on inside-out tracking even while it was being made in other important areas of AR like display technology. Inside-out tracking has remained for years an elusive key to the success of AR, but it seems like Microsoft has made a game-changing stride with HoloLens.

It seems many have been too distracted by the headset’s all too narrow field of view to realize how jaw-droppingly good the device’s inside-out tracking is. It was so good compared to anything else that I’ve seen that I actually waited until a second time to get my hands on HoloLens to double-check what I saw initially. On that second check, it was confirmed: rock solid inside-out tracking. The kind that’s so good that you can shake your head back and forth much faster than you’d normally have reason to, and see that the augmented objects in the environment appear locked in place; the same kind of tracking quality you’d experience with top outside-in tracking systems like the HTC Vive and Oculus Rift. HoloLens’ inside-out tracking is better than anything we’ve seen from other big names in AR, including Magic Leap and Meta.

SEE ALSO
Daqri is Now Shipping Its AR Smart Glasses to Professionals

The video above gives a good idea (but actually looks slightly less impressive than what it looks like through the headset) of how solid the inside-out tracking is; the virtual screens stay nearly locked in place as the user moves their head.

The question of robustness (the ability of the tracking to work consistently in a wide range of environments and lightning conditions) still remains open. I’ve only had a chance to test HoloLens in a select number of environments, but one of those places was a veritable torture-test for any inside-out positional tracking system: on a busy conference show floor.

samsung gear vr sdc demos
See Also: Oculus CTO Affirms Positional Tracking Priority for Gear VR

It was several months ago, my first time getting the device on my head, in the middle of the show floor with people buzzing about, walking directly passed me. I popped HoloLens on my head and placed a few floating windows around the area.

Normally I would expect all the moving people to seriously throw the system off; it has to look at certain parts of the environment and somehow tell what’s moving and what’s static, even though to a camera the whole world appears to move when you move your head. This is a fundamentally challenging computer vision problem. But in my time with HoloLens, it easily ignored the people walking all around me, and kept the windows and other AR objects stuck convincingly where they were supposed to be. So things are looking good for robustness.

Another reason there hasn’t been much buzz about HoloLens’ game-changing inside-out tracking may be because it wasn’t always this good, crucially back when Microsoft first introduced the device to an initial swath of press back in early 2015. According to a source working with the headset, the tracking has indeed improved with time.

SEE ALSO
Microsoft to Expand HoloLens Availability to 29 More European Markets in December
HoloLens Sensor/Camera Cluster. I'm counting 11 lenses and sensors(!)
HoloLens’ sensor cluster has a whopping 11 elements

While the tracking on HoloLens is outstanding, the device falls short of being a total breakthrough in AR due to the diminutive field of view and clunky gesture input. A next-generation HoloLens which fixes those criticisms will blow the lid off of AR as it stands today.

AR is isn’t the only thing to get excited about when it comes to inside-out positional tracking being ‘solved’; virtual reality too would massively benefit, opening the door to untethered virtual reality headsets with huge (potentially limitless) tracking volumes. It’s what mobile VR headsets like Gear VR (which lacks positional tracking) have been eagerly waiting for.

If Microsoft’s inside-out tracking success can be replicated, the next step is making it affordable. With the HoloLens development kit currently priced at $3,000, there’s a way to go, but the critical problem of inside-out tracking may finally have a solid existence proof.

This article may contain affiliate links. If you click an affiliate link and buy a product we may receive a small commission which helps support the publication. See here for more information.


  • dogtato

    That’ll be super cool to have with a decent FOV. edit: and sub-$1k

    Also, the Vive uses inside-out tracking, it’s just more selective about what it looks at.

    • It’s not selective what it looks at, it can ONLY ‘look at’ the laser pulse sweeps from its specially designed and placed base stations. Comparing HoloLens (an entirely independent, computer vision based, inside out solution) and Lighthouse is, frankly, bizarre.

      • Agreed! HoloLens doesn’t require any change to the environment. The environment is the reference. Almost any environment (save reflective and light absorbing materials).

      • dogtato

        Calling lighthouse outside-in is bizarre. The lighthouses don’t track the headset. I don’t disagree that Hololense tracking is on another level.

        • psuedonymous

          “Calling lighthouse outside-in is bizarre.”

          It is, however, correct. Check the inset box on P30: http://www1.cs.columbia.edu/~drexel/CandExam/Motion_Welch_Foxlin.pdf

          Scanning laser positioning systems like Lighthouse and iGPS are outside-in.

          • dogtato

            Thanks for the link. It does seem to be both bizarre and correct.

            It seems like it’s classified that way because of how it performs and not by which direction it’s looking.

          • brandon9271

            Read it again. Lighthouse is inside out. The sensor are on the objects being tracked and looking out. The problem is people use the terms ‘sensor’ and ’emitter’ interchangeable when they are completely opposite. For example, the Wii sensor bar wasn’t a sensor at all. It was an emitter. The sensor was actually the camera in the remote. Thus the Wii and Vive are inside looking out. :)

          • Victor

            Well, we use the terms absolute/relative tracking combined with active/passive. The Vive uses an absolute (lighthouses) and active (sensor on tracked body) tracking. The Hololense does SLAM, thus it uses a relative tracking. Older HMDs with inertial sensors also used relative tracking. Our CAVE system has static cameras and passive markers on the tracked bodies, thus it is a passive absolute tracking. Does this make sense? :)

    • Milosx

      Hololens is intended for more professional use and therefore does not
      require full immersion. User is not supposed to forget where they are,
      as with VR. Games are still great on it and you quickly forget you are
      looking “through a window” especially with a great story such is
      Fragments.

      • D.L

        Let’s be real here. The low FOV is a technical limitation, not a practical one. More FOV is always going to be better, no matter what you’re doing.

    • benz145

      Inside-out/outside-in doesn’t really have to do with ‘what is looking at what’ (even though it certainly sounds that way if you’re thinking about a sensor as an ‘eye’), it’s just about whether or not the system relies on external components to aid the tracking. Really just a matter of semantics though; we could (and often do!) call inside-out tracking ‘standalone’ to try to aid in clarity.

      • brandon9271

        I think “tracker-free” would be a better term because in nearly every case I’ve seen the terms used, inside-out or outside-in has everything to do with “what’s looking where” :)

        • benz145

          If we’re nit-picking then the Vive would fall into “tracker-free” too because the Lighthouses aren’t trackers, they’re emitters ; ).

          Vive’s unique method makes the colloquial definition of ‘outside-in’ seem not to apply, but it’s a term which has been adopted by the industry and is understood to mean any system that uses a static component that’s external to the tracked object (“tracked” in this usage doesn’t mean the object is being “watched”, it means that its position is being measured).

          • brandon9271

            Well then that would mean Vive is still inside out because the objects being tracked are the lighthouses themselves, because as passive devices the lighthouses aren’t actually tracking anything :) Again, it’s the same as the Wii remote example. The sensor bar wasn’t a sensor. It was a passive device. Two IR LEDs that were actually being tracked by the Wii remotes. Again, inside-looking out tracking, where what’s actually being tracked is the IR leds. It was only called a “sensor bar” so as to not confuse consumers but it’s 100% incorrect.

  • VRgameDevGirl

    Geez. The dev kit is 3k I wonder how much the consumer edition will cost .

    • John Roberts

      I’d gladly charge 3k on my credit card if it had complete field-of-view and looked like when Microsoft previewed minecraft and the golf tournament on stage.

    • OgreTactics

      There won’t be a consumer edition, this isn’t a consumer technology yet.

      • MdM

        “Yet”. Kinda implies that eventually there will be a consumer edition, right? Not to mention the numerous statements Microsoft has made indicating that this will become a consumer (or, at least, enterprise) product.

        I’d be surprised if the consumer edition wasn’t around the $500 mark. It’s expensive to make these now, since they aren’t mass produced. Plus, the AR/VR market will drive down necessary component prices quite a bit by the time Microsoft releases something. They’ll likely find ways to subsidize the cost some as well.

        It’s gonna be hard to get consumers to buy yet another device that they feel compelled to upgrade every 2-3 years. Historically, people have kept PCs for quite a bit longer than this. The tablet market is an indication of this struggle, although the value added for an AR device is quite a bit more than for a device that essentially amounts to a large phone. That’s one thing that the Gear VR type solution has going for it… you don’t have to pay for another expensive device. Can’t wait to see what Google has to announce next month…

        • MosBen

          This. There isn’t a consumer model BECAUSE it’s $3k. When they go through another iteration that makes the tech better and get production facilities up and running the cost will come down significantly. My guess is that around the time we’re talking about the next generation of VR HMDs we’ll be talking about a consumer version of Hololens.

          • OgreTactics

            What do you call the “next generation of VR HMDs”. Because when I say it’s not a consumer PRODUCT, I mean it won’t be until 5 to 10 years in the future.

            Not just because of the price, but because of the technology. Everybody who has tried HoloLens more than twice will tell you: this is a nice experimental device, but this isn’t something anyone would use in any real-life situation (except maybe engineers and technicians working in big corporations), because of the way too narrow field of view and limited resolution/opacity.

            And this, won’t get fixed or better until 5 to 10 years in the future, for AR glasses to even match current VR HMDs specs. And if you’ve followed, just the fact that HMDs are more than 500$ and limited FOV/Resolution have put them very far from their original median sale projections.

          • MosBen

            What makes you think that it’s so far out? A lot can happen in 3-5 years, which is when I think the Rift 2, etc. will come out. 3-5 years ago we weren’t even talking about VR as a consumer product, and now here we are. If the biggest limitation on the FOV for Hololens is their processor, which from what I’ve read it is, then 3 years is plenty of time for them to develop the next generation chip that would allow for a larger FOV, and it’s also plenty of time to get production facilities of sufficient scale to bring the cost down.

          • MdM

            MosBen, I totally agree. Never before have we seen so much money being poured into all facets of an industry that barely exists in the consumer market. Due to this, a LOT will happen in the next 3-5 years.

            The limited FOV of the HoloLens and the low resolution of current VR HMDs are the EXACT same problem, right? The solution lies in increasing pixel density and designing CPUs/GPUs that can drive a higher resolution screen and render desired video. We are on the verge of having a 4k solution, and I imagine in 5 years, we will have an 8k solution and be on the verge of a 16k solution. If Microsoft, keeps the same perceived pixel density, this will allow them to cover the entire FOV.

            Note that it’s one thing to be able to push that many pixels out of a GPU, but it’s quite another to be able to render compelling imagery at higher resolutions. The latter has been the factor that historically set the console gaming generation cycle time. However, HMDs are about to take a giant leap forward with this by utilizing foveated rendering. This will bring about a huge leap in performance for both mobile and PC tethered HMDs.

            Augure, you are a very confusing person to converse with:

            “There won’t be a consumer edition”
            “I don’t think they will be ANY consumer HoloLens”
            “…when I say it’s not a consumer PRODUCT, I mean it won’t be until 5 to 10 years in the future.”

            So, do you think there will eventually be a consumer HoloLens or don’t you???

          • MosBen

            Exactly, the whole point of the article is that MS’ current hardware has a good solution to the long existing tracking problem, but it’s hard to do, so the sacrifice was a limited FOV. In 3-5 years we’ll have significantly better and cheaper hardware, so it’s reasonable that it could get to a spot where it’s good enough for a consumer device. Again, the whole point of the article is that Hololens’ technical solution works, it’s just limited by the strength of the hardware. I don’t know why that’s a problem that takes 5-10 years to solve.

          • OgreTactics

            “What makes you think that it’s so far out?” Science.

            Nothing to do with processor, we are actually very limited at how far we can go with the FOV/Resolution specs of what are essentially transparent lightfield displays (this is very advanced!): these are not just screen or piece of glasses on which we project, there are transparent glasses with micro-lenses arrays, like VR lenses but for each set of screen pixels or projected pixels.

            That’s why I say, there’s no difference between VR and AR in terms of device: the only thing that is usable now are Virtual Headsets, which as of now can have between 100°-150° FOV and 2K/4K resolution, and it’s a baffling non-sense that they don’t integrate the unique components set that can both do external viewfinder, environment tracking, AR, head and body tracking, hand tracking, interface tracking (like the Vive pans that you see inside VR and changes buttons) etc…in fact going back to my OP I think this is one of the biggest problem setting back Virtual Headset, they’re an unfinished product paradigm (if you look at my comparison between Palms and Iphone you’ll understand).

            Virtual Glasses, while they are being experimented with the HoloLens, Meta 2 or even the ODG R7, are FAR from being usable consumer products that even matches the current GearVR specs, because the thing is that the technology is far from being ready if you look up lightfield glass screens, we don’t even know managed to more than 40° FOV, sub-720p resolution and full opacity in these first Virtual Glasses.

          • Chris Bordeman

            “There isn’t a consumer model BECAUSE it’s $3k.”

            No, dolt, the consumer edition doesn’t exist yet. Like other similar products released recently, the dev edition is far more expensive than the eventual consumer release.

        • OgreTactics

          I don’t think they will be ANY consumer HoloLens, except for wealthy engineers and technicians.

          The reason why is because AR/VR/MR differentiations are bullshit, and at a certain level things are lead by what’s possible in science, not product design or marketing/pricing.

          While it’s great that Microsoft and few companies gave a shot at Virtual Glasses now, it’s nowhere near a consumer technology. And as I stated, the differentiation between AR and VR is bullshit: there is no difference in paradigm between a tracked environment on which virtual objects are partially super-imposed and a full (tracked or not) environment on which a whole virtual scene and object is super-imposed.

          The difference is between VR Headsets, which means that we have lenses pointing at a split screen that by default can only display full environment but with tracking cameras become “AR” enabled, and is feasible and accessible now although not perfect, and VR Glasses which means that we use very advanced micro-lenses glasses with transparent pixels or projectors to display images that could very well be independent from the environment but usually are tracked by cameras, and is very limited in terms of tech for the time being and the few years to come.

          So no HoloLens is not a consumer product, it’s a very advanced prototype of a future consumer technology, but right now we’re barely capable of having advance Virtual Headset that are affordable and tracking cameras should already be implemented not just because it’s the best way to do “AR” right now, but also because it also requires environment tracking for head and body tracking, if hand tracking. VR/AR are the same thing, and people should focus on headset right now.

    • Chris Bordeman

      Dev kits are always far more expensive than the consumer products.

      • VRgameDevGirl

        depends. Oculus dev kits were less than the consumer model.

  • David Mulder

    How would you say this compares to the inside out tracking provided by Tango? Just kinda curious, as that’s a more affordable set of hardware by which I have been quite impressed as far as the inside out tracking goes.

    • In terms of quality, it seems to me that they are very similar. They both use hardware acceleration to reduce lag. It would be interesting to know how much power each one requires. It’s an important detail when talking about battery powered devices like smartphones and AR HMDs.

    • OgreTactics

      Seemed to me that HoloLens tracking was way more efficient and precise than Tango.

      Tango is lightweight but can be laggy, especially when you try complicated tracking algorithm for hands for example. VSLAM mapping (which is not photogrammetry despite some say) of an environnement is a usually a good way to compare the efficiency of depth/tracking sensors.

      • DiGiCT Ltd

        The tracking speeds will improve in future for sure, it is mostly based on hardware, i had those expierences with several sdk for AR on android, using pads and phones, even different devices.
        The result was they all performed different with the same app, just different hardwware in it.
        Current camera’s are not perfectly designed for tracking, it takes more time to get those working out better in AR.
        Antoher part is the lighting conditions that influence a lot too.

      • benz145

        My thoughts as well on the tracking; Google has yet to really poise Tango as a solution for VR. The writing is on the wall, but so far they acknowledge it isn’t quite ready for VR.

        This is essentially the latest we’ve heard on that (though quite old now):

        http://www.roadtovr.com/google-wants-to-use-tango-tech-for-vr-but-admits-current-dev-kits-arent-optimized/

        But that’s their stance for now until they’re ready to push it as ready for VR.

  • Dan

    Heck yes inside out tracking is where all tracking will eventually lead to. This is probably what their custom chip is for. Now if they could also add a complete blacking out with lcd behind the screen, fix the res and fov, add a connection to the gpu, and put the chip in a couple of hand controllers they’ll be in the current vr race and kicking everyones butt. Theres no fight between vr and ar, some people want one, some the other but most people will want both, like wanting a computer that can do at least 2 things.

  • Tried HoloLens yesterday… and I’m going to write a little review about it, too. Agree with everything… but: you say that Hololens is better than Magic Leap tracking. Have you tried it?

    • Adrian Meredith

      There was definitely some small jitter from the leaked videos of magic leap but I’m sure that will be resolved and it’s more than made up by the vastly superior image

      • lovethetech

        All ML videos are now established as created by Movie CGI producers like Weta and as fake videos.

    • benz145

      Only what’s been released publically.

    • Milosx

      It seems no one used Magic Leap. Those videos could have been made in mocha, boujou or any other software for 3d tracking, and just added a bit of jitter for perceived small imperfections. They announced it years ago, so why is the dev kit not ready yet?

  • OgreTactics

    Quick Reality Check: HoloLens is an experimental prototype that will only become a consumer product in 5-10 years from now.

    When you look at the tech behind, It’s impossible for VR Glasses (fuck calling things AR/MR) to be wide FOV high-resolution right now.

    But solutions like this HoloTracker, Intel RealSense or Google Tango, are the next priority key to make HMDs more than gadgets. They do everything from motion tracking, environment mapping and scanning, AR, hand tracking, device and interface tracking (like the Vive pans changing buttons)…

    And now that they managed to make it small enough to include it in laptop, or even in fucking smartphone, what the heck are HMDs brands waiting for to integrate it?

    • OgreTactics

      Addendum: this is as far as we can go that I’m aware of (110°) but this doesn’t deal with resolution and opacity of image: https://vimeo.com/102459256

    • DiGiCT Ltd

      Yes it is going to take many years before getting a good AR HMD.
      Metaio also has one cheaper but similar specs and issies as hololens.
      The required components for a perfect device are non existing yet or even might be way too expensive to manufacture atm.

      AR hardware component cost exceeds VR, as it is more complicated if you want to do it good.
      The best results in AR i actually get by Using a PC or notebook with a HD USB camera.
      You could try it and you will see stuff goes much smoother and faster.
      Performance is an issue on those HMD’s, making fast motion and movement causing tracking issues.
      If you look very clear at all MS hololens videos, the users motion are in general linear and slow, no very fast action and spinning aroound 180 degree, as that is simply not doable atm.

      • OgreTactics

        I think the intermediary, if not the REAL solution is that AR will really take-off when it is integrated into VR HMDs.

        In fact I don’t make a difference between VR and AR: you have a visual interaction device (yesterday it was screen, now it’s HMDs, then it’ll be glasses and lenses) and you either display elements on your environments (which we now call AR) or your surround your environment (which we call VR).

        • I agree

        • DiGiCT Ltd

          I think AR eventually will replace phones and computers, where VR will be replacing the imersion games and movie expierences.
          As both tech only actualy have 1 difference which let you expierence it different.
          It is being still in reality or totally closed of from it.
          Some situation you would rather be out of reality so VR fits better, but in most cases you would like to be in reality in which case AR will ift.
          Thats why IMHO AR will be in furture much bigger.

          • NRGuest

            VR isn’t likely to completely replace games and film. The only games that VR really works for are 1st person games. RTS, platformers, 3rd person games, etc are all likely to stay on screen (although maybe an AR screen).

          • DiGiCT Ltd

            dpends on how they are designed, as we are working on a game like xortex from the lab, it is a kind of 3rd person game, but if you say gamepad 3rd person games i could more agree on it.
            On the other hand RTS games can be really fun in VR and even more exciting too, we are prototyping that atm.
            After doing the spectator mode in Dota2 I got inspired to seee an RTS game like that in VR.
            At the end it is really up to how it is designed and I am sure plenty of games and movies will be on flatscreens for a while too as you say.

        • Dave

          Actually I see it going in the other direction, making AR more portable and lighter. It needs to step away from any notion that it can run games and concentrate on other comerical uses. Something like the ODG R-series smart glasses is more likely the way forward for AR.

          • OgreTactics

            Nope. This is the way forward…8 years in the future (which is early considering…) at which point it’ll reach current VR HMDs specs, and the point will be both AR and VR since it’s just a matter of FOV (and opacity). But ODG R-series is damn impressive in it’s current state.

          • K

            If it plays game, why step away from that?

    • K

      ”that will only become a consumer product in 5-10 years from now.” 5 yrs from now, or 5 yrs from lanuch. Didnt microsoft people say around the time of launch that it is a five year game?

      • OgreTactics

        I was wrong, now that I know more about lightfield screen technology: this is a 10 to 15 years in the future game. Which is WAY too far to justify any bullshit product.

  • psuedonymous

    The tracking is good enough for Hololens, but that is due to the very low FoV (30°-35°) and moderately low (60Hz) refresh rate. It at least matches other lab systems and operates in the real world, but it damn well should do using an array of multiple different types of camera AND a dedicated custom coprocessor!

    But that ‘good enough’ is not even close to good enough for even todays VR, let alone future HMDs. The narrow FoV increases tolerance for tracking hiccups and inaccuracy; this can be seen by injecting tracking jitter and offset to a VR HMD, then reducing the FoV. The lower the FoV, the higher the tolerance for bad tracking.

  • Srinivasan

    In a populous country such as India, with all sections of people living in, with several issues that come up / presently existing… like for example, poverty, malnutrition, health issues, what are the probable use cases of Microsoft HoloLens

    • Bryson

      You will get to see Virtual food on the table and get to look at virtual toilets. Just dont try to use them

  • Chris Bordeman

    “the next step is making it affordable. With the HoloLens development kit currently priced at $3,000, there’s a way to go”

    What an imbecilic statement, since dev kits cost has nothing at all to do with consumer pricing.

    • Tony Murchison

      The DevKit price isn’t random. True, consumer products are virtually always cheaper than their development counterparts, but that’s the result of a lot of optimisation and scale improvements.

      It doesn’t say that the price is going to be ludicrous. It says that Microsoft will have to do a lot of work to improve it. Which is true.

      • K

        ”True, consumer products are virtually always cheaper than their development counterparts” Usually…how cheaper compared to development counterparts?

  • Dave

    No one’s talking about it because it’s can’t run any decent games, very limited capacity running comercial games. The capabilities for the device will have to grow considerably for this to become anywhere near it’s potential – right now you’re looking at 10 years at least but probably another 15 until game companies get on board.

    The comments on the technology is laughable. By the time it’s has any commerical use the technology would have been overtaken in other VR/AR devices if it hasn’t been already.

    Has practical purposes yes, can create inovative game content yes, will entice the gaming studios no. Is exciting no. It’s dull, limited and expensive.

    Sorry but all it really is, is a samsung gear with a clear screen.

    • K

      It doesnt need to run ‘decent’ (i presume you mean console/PC quality games). It doesnt need to. All it needs to do is to run AR games that integrate with our surroundings and objects (the kind of games shown in the demo videos), for people to get excited. But with a greater FOV and offer better realism to the games and also various kinds of inputs/complete handtracking etc of the games.

      • Dave

        Yes but you understand right the problem is with the text “but with a greater…”. The hololens has already been superceeded in many areas and in it’s current guise it will be incapable of causing a stir in the AR market especially if the developer price is anything to go by. AR aside currently you’re getting a much better experience with a VIve at a fraction of the cost and companies like Meta and ODG are producing more exicting AR devices. So my point stands “No One Is Talking about It” because it’s largly uninspiring, unexciting, expensive and already obsolete. The only chance the Hololens has is in the commercial sector but it needs significant improvements to make it there too.