At CES 2017 Qualcomm unveiled its Snapdragon 835 processor and along with it a solid focus on the mobile VR market. We got a chance to test that new focus by going hands on with Qualcomm’s latest reference hardware and find impressive inside-out tracking using just the phone’s onboard sensors.

Qualcomm’s processors and chips power perhaps the majority of smartphones today, and the company often creates reference hardware designs to demonstrate and provide guidance to their OEM partners when developing these next generation devices. At CES 2017, the company showed off a mobile VR experience powered by reference hardware sporting their latest Snapdragon 835 platform.

SEE ALSO
Qualcomm Positions New Snapdragon 835 Mobile Processor Toward AR and VR

The experience featured a tie-in with the new Power Rangers movie, due for release this year, and while we could talk about that, it’s the tracking technology that deserves the most attention. I was able to try it out, and I can say that it may, potentially, represent a breakthrough development for the VR industry.

The demo experience was prominent at Qualcomm’s space on the CES show floor, but anyone walking by may have been fooled into thinking that the experience was primarily about Power Rangers, not realizing that a hugely important development in technology was also being demonstrated. The company had several small demo areas, each complete with a VR headset and each with space enough to take a few steps around whilst inside the experience.

The headset units themselves comprised the reference Snapdragon 835 phones mounted inside third-party Cardboard-style “passive” headsets. From the front of the headset you could see the reference phone’s rear camera sensors. We have no details on these just yet, but as you can see from the images, the phone is packing more than just a single RGB sensor. We’d hazard a guess that, similar to Asus’ ZenFone AR, also announced at CES this year, Qualcomm’s reference device also sported additional depth sensors.

SEE ALSO
Asus 'ZenFone AR' Google Tango, Daydream VR Phone Launched, Specs Revealed

Putting on the HMD and headphones, there was also a focus knob that the headset, pretty standard stuff. Paying little attention the content at hand, I began looking around to test the positional tracking. The view changed as my brain expected it to. Then I began quickly moving and leaning my head around to further test the system, and it managed to keep up, although perhaps not as quickly as systems like SteamVR’s Lighthouse or Oculus’ Constellation. It felt very slightly behind in positional tracking, but for rotational tracking, it was just as good as any of the PC VR headsets.

qualcomm mobile vr inside out tracking ces 2017 (2)I explored as much and as far as I could within the small demo area, and it held tracking throughout the entire volume, from standing to crouching. There was a railing that was slightly reflective, so I tried going really close to it in an attempt to freak out the tracking, and it stayed stable until I was around a foot away, where it began to hiccup. Throughout the whole demo however, there were only a few instances where the view would drift a touch or perhaps became “floaty”.

Knowing that it this was inside-out tracking powered by the phone’s front-facing camera sensor array, I tried waving my hands around the front of the headset, yet the view stayed stable for the most part, right up until I put my hand right in front of the camera to keep it from seeing anything. Unsurprisingly, my virtual world view would stay stable for a second before positional tracking was lost completely.

Despite some flaws, the tracking performance demonstrated by Qualcomm was extremely impressive, especially considering that the entire positional experience was tracked by an onboard camera array, all powered by a mobile phone. Overall, the quality of the tracking felt similar to that of Sony’s PlayStation VR (PSVR). But unlike the PSVR, and most other VR headsets with positional tracking, you don’t have to mount external cameras, or laser base stations. What’s more, have could have a potentially unlimited play area to walk around in.

qualcomm mobile vr inside out tracking ces 2017 (3)The positional tracking forms part of Qualcomm’s reference design, so partner OEMs utilising the company’s SoC’s (System on Chip) will have the ability to take advantage of it. This means that we may very soon see this level of positional tracking in mobile VR headsets at last. More than that though, Qualcomm’s advances here may also have implications for other areas of virtual reality, augmented reality, and beyond.

All of that said, my demo was (as ever) best case scenario and we should bear that there may still myriad other limitations. How will the system deal with differing lighting conditions and environmental make up. We don’t have definite knowledge of how reflective surfaces and different kinds of light in the environment may affect tracking. However, talking with a representative at the booth, solid colors and reflective surfaces were in fact cited as problematic. To what degree, we’ll have to see, but the representative did say that the system detects edges and corners. Further, we don’t yet know how demanding this kind of mobile-powered computer vision is and how quickly it would drain battery life.

Still, even with some these potential limitations in mind, it’s possible that this development could be considered a real breakthrough. If it really does work this well and OEMs are quick to adopt (as well as willing and able), then we’ll get the fabled positional tracking we’ve always hoped for in mobile devices quite soon. If rumors are true, then we may even see it powering the Samsung Galaxy S8, due for release later this year.

Newsletter graphic

This article may contain affiliate links. If you click an affiliate link and buy a product we may receive a small commission which helps support the publication. More information.


  • Sponge Bob

    Are you putting your Lighthouse in a garbage can ? Not yet ?

    Can you test it on an open floor without any objects nearby ?

    • Get Schwifty!

      There will be a range of tracking solutions in use, it’s probably not going to be limited to Lighthouse or Constellation by any means…

      • Sponge Bob

        well. if one solution is clearly better then why use other solutions ?

        the real question is to how to define the term “better”

        • beestee

          It is not which of all solutions is overall better performing, but rather what each solution is better at doing.

          Say you want to use VR in a space completely mobile and un-tethered. Lighthouses will not work at all in this situation, despite providing a superior fidelity of tracking when they can be set up properly.

          Or should we all stop using mobile phones, tablets, laptops and even desktops in favor of servers, since they are overall a better solution for storage and processing?

          • Sponge Bob

            What is “completely mobile” ?

            Like walking down the street in your VR headset ??

            I don’t think it would be a good idea . LOL

          • MosBen

            As with listening to music and watching Netflix, it’s entirely likely that with VR people will own systems that are technically superior but limited to home use as well as systems that are technically inferior but able to be taken out into the world relatively easily. We don’t need to discard one in favor of the other, and developments in one space often lead to benefits in the other space.

          • user

            it might be hard to understand if your apartment is only 1 room and you dont have friends to go to or you never go to other places for work or vacation.

          • VR Dev

            Inside-out tracking will be the only system in a few years from now. Also, VR will only be a special case of AR. We will all have AR glasses and clip-ons to cover the real world to get VR. Cameras on the glasses will handle positional tracking – not external cameras.

          • user

            then you might want to tell me how every physical object that you want to bring inside the virtual world will be tracked

          • Sponge Bob

            with 8 cameras on a headset facing in all directions ???

            maybe 5 360 degree cameras – on each side of you head except the bottom – unless you are decapitated :)

          • Sponge Bob

            “Cameras on the glasses will handle positional tracking”

            Perhaps you should know that IR cameras do not work outside, in bright daylights
            Are you suggestingt hat AR will be like VR of today with some dorks in monstrous headsets playing VR games in dark spaces ? Then what’s the point of AR ?

            You are also suggesting that inside-out will prevail on mobile VR ?
            So mobile phones with limited compute power will run full inside-out ON TOP of rendering VR ????
            ArE YOU CRAZY, DUDE ???

          • Sponge Bob

            by IR cameras I meant cameras not for inside-out head position tracking but for controller tracking
            None of those inside-out dudes can explain how they are gonna track those controllers – indoor and outdoors
            Hah ???

          • user

            “Qualcomm Technologies designed the processor to efficiently distribute computer vision workloads, particularly the 6 degrees of freedom—or “6DoF”—positioning used in Tango and other AR applications. Snapdragon processors distribute this processing across the Qualcomm Hexagon digital signal processor (DSP), CPU, and Qualcomm Adreno GPU. While the Hexagon DSP helps determine exactly where and how the device is moving through 3D time and space, the CPU and GPU are free to handle displaying the enhanced visualized 3D information on the screen and running other processes needed to deliver an immersive Tango user experience.”

          • OgreTactics

            This, except we will never had AR Glasses, first “Virtual” headset with AR as it’s main purposed and it’s full spectrum which is VR, and “Virtual” glasses in 10-15 years if VR manages to pick up now.

          • beestee

            Hmmm. If it could augment perceptive capabilities then why not?

            Human error in judgement is pretty poor as is.

            Maybe pedestrian related accidents would decline if we had a head-up alert system to warn us of hazards while walking down the street? Just because the dangers are there it doesn’t stop people from distracting themselves with their electronic devices in situations where they should be paying attention.

          • Sponge Bob

            I was talking about VR – not AR

            although they will merge sooner or later hardware and software-wise, these 2 modes are separate and will stay separate

            I repeat: I do not see any huge value proposition to using inside-out tracking for “mobile” VR as opposed to using external tracker

            Because VR just can’t be “too mobile” – you will always stay in some more or less confined physical space – not walk down the street in real world experiencing full VR immersion

          • Malkmus

            Mobile will likely become a very appealing and affordable alternative to PC VR when it gets 6DOF tracking, at least for people who already have phones ready for it and only need to drop extra money on a GearVR or Daydream HMD supporting it.

  • wheeler

    Sounds like they’re making good progress but still have a ways to go.

  • Deshawn

    “Putting on the HMD and headphones, there was also a focus knob that the headset, pretty standard stuff.”

    I’m not sure why you would consider a focus knob ‘standard’ the Gear VR and PSVR are the only headsets with a focus knob.

  • OgreTactics

    There is NOT one actual Virtual Headset on the market until this is integrated. The only realistic challenges it brings are how to correctly convey see-through AR with accurately disposed cameras and micro-lenses angles, and hand-tracking buffering (which as demonstrated on Tango, does work for general movement and thus touch-screen like interactions but is not accurate and powerful enough to model, reconstruct and track hands in rt).

    “We don’t have definite knowledge of how reflective surfaces and different kinds of light in the environment may affect tracking”.

    I beg to differ: http://people.csail.mit.edu/yichangshih/mywebsite/reflection.pdf
    and http://stanford.edu/class/ee367/Winter2016/Hu_Report.pdf

    • Sponge Bob

      It is well known fact that bright daylight makes IR-based (depth or regular) cameras useless, or at the very least severely degrades their accuracy

      Active IR time-of-flight measurement by depth cameras outside in direct sunlight ?

      Good luck :)

      And what is AR for ?

      • OgreTactics

        Weren’t you the one saying that imagining people just going out with VR headset is completely wack (which I agree with)? Direct sunlight is not problem.

        As for AR…man…I underestimated how much more important than VR it is. In fact it doesn’t make much difference with VR since it’s just the continuum of a spectrum of more or less overlaying of your FOV with object, walls and full scene. But because of that bullshit mobile AR like Pokemon Go, and these bullshit AR glasses which are cool but nowhere near operational technologies, people seem confused.

        The point of AR is NOT in seeing your environment and putting virtual things in it, it’s about tracking the environment, it’s object, it’s movements accurately enough so you can transform it. Also this means the REAL, professional, creative and social use of Virtual Headsets and later on Virtual Glasses/Lenses: replacing all screens virtual objects (which means you don’t need physical screens anymore, or even windows and frames in UI) or scenes AND controllers like keyboard/mouse/pad with real hand and voice interactions. This is the global vision of an actual virtual headset that there’s a point to adopt, buy and use everyday, not these “video game” unfinished gadget that are Oculus/Vive/Gear.

        • Sponge Bob

          “AND controllers like keyboard/mouse/pad with real hand and voice interactions”

          Slow down, cowboy :)

          none of the touchscreens or voice assistants or whatever can beat good old fashioned wired computer mouse – the good one will cost you like 100-150$

          How is direct sunlight is not a problem if it kills all the IR sensors, including controller sensors ?
          They’ll have to figure this out to have truly mobile VR/AR

          • OgreTactics

            1. I don’t want a mouse for anything, and I just want a keyboard to type nothing more. They have existed for respectively 40 years and a century (type writer). So I think it’s about time to get rid of it for something better.

            2. Well as of now, VR will not be used outside of appartments and offices, in which there are almost no situation of direct sunlight exposition.

          • Sponge Bob

            1. the point I was making is that it’s pretty damn hard to replace mouse and keyboard
            touchscreen is only partial replacement, and it doesn’t work too well with VR/AR anyway
            finger tracking e.g. from LeapMotion ? – sucks in my view
            voice input ? – good luck relying on it for keyboard replacement

            2. What about AR ? Being outside in the real world is the main AR attraction

          • OgreTactics

            1. Well I see the paradigm this way: For 40 years a century, the interaction we’ve had with the infinite possibilities of computers, were single inputs buttons, then later on a pointer (literally a point) moving on flat limited surface. Something is wrong is we’re still doing this. Touch screens aren’t shit. They make 4x more sense than keyboard+mouse+screen, but not 100x more. Hand interactions extended by VR interface gives a more power than just this already advanced tool that is the hand, and voice is the way we note, express and communicate. Now, Leap Motion is cool but meh, and voice A.I. are meh…doesn’t mean that in their potential they already make 100x more sense than previous controllers. The only to figure out is of course feedback. This is the only point of a keyboard. Have you heard of Tenvas: http://www.youtube.com/watch?v=LrbJWiAc6JI

            2. I responded to that in my other comment.

  • Albert Hartman

    The 835 demo convinced me 6DOF inside out tracking was going to be possible in mobile eventually. It also unfortunately convinced me that Vive/Oculus class imagery and refresh rates are the minimum that will create acceptable immersion. Unfortunately 835 falls far below.

    • Sponge Bob

      Which brings us to this fundamental conclusion:

      In order to succeed in the near future, mobile VR should ditch inside-out tracking in order to concentrate scarce computing (and battery power) resources on what VR is really all about – hi-res rendering with enough FpS to give you sense of complete immersion

      • Albert Hartman

        I’d even accept a pocket-carried brick connected by a 1m wire to the HMD if it would allow a fast enough GPU to be used. 90-120 fps with sufficient triangles is a real deal-breaker for mobile VR immersion.

        • Sponge Bob

          90 FpS is not the main problem – the problem is that you have to maintain that 90 fps WHILE increasing resolution… from full HD to 4K then further
          Each time linear resolution doubles the GPU compute cycles quadruple
          AND on top of all that they want to run SLAM 3D codes in real time, all the time ? On mobile battery-powered device ?

          gotta be kidding

          • Albert Hartman

            Right now people doing it with battery-powered laptop worn as a backpack – nutty. I was hoping for a battery-brick sized unit that fits into your pocket, maybe 5″x4″x1.5″. You could fit 60 watt-hour battery into that pretty easily. It could power a meaningfully powerful GPU and CPU for several hours.

          • Sponge Bob

            what about PC GPU rendering then streaming over wi-fi – TPcast or something ?

            looks like the only practical way right now
            you want to be untethered, yes, but you play in one room only
            – can move your PC setup to another room if needed
            sucks though