HTC has announced a new set of tools allowing developers to build applications which take advantage of the Vive Pro’s stereo front-facing cameras, effectively turning the device into an AR headset dev kit. The new tools allow the cameras to capture depth, spatial mapping data, hand input, and seamlessly shift between VR and AR worlds.

While the original Vive launched with a front-facing camera, it went sorely unused. This time around, with the Vive Pro, the company is offering the VIVE SRWorks SDK. Announced last week, HTC says that the SDK includes three modules: a depth module, see-through module, and a 3D reconstruction module—effectively a foundational set of tools enabling the headset to sense the world through its front-facing cameras, and allowing developers to use that data for creating interesting experiences that can be pure AR or VR, or a combination of both.

SEE ALSO
HTC Vive Pro Headset Review: Welcomed Improvements Overshadowed By a Steep Price

This example video, using the Vive Pro and the SRWorks SDK shows how the tools can be used to seamlessly link VR and AR worlds, creating new and interesting gameplay and application possibilities:

The company says that the SRWorks SDK includes support for native development with plugins for Unity and Unreal, and that the modules can enable the following:

  • Depth
  • Spatial Mapping (static and dynamic meshes)
  • Placing virtual objects in the foreground or background
  • Live interactions with virtual objects and simple hand interactions

This example shows how the Spatial Mapping module can create a model of the room’s geometry for use in applications:

The Vive SRWorks SDK is available in beta through the company’s developer portal.

Like the ZED Mini depth camera add-on, thanks to the new tools, the Vive Pro’s front-facing stereo cameras effectively emulate the sort of experience that AR glasses will ideally achieve in the future—an immersive, wide field of view with precision tracking and environment mapping, potentially turning it into a great dev kit for developers building toward XR hardware of the future.

This article may contain affiliate links. If you click an affiliate link and buy a product we may receive a small commission which helps support the publication. See here for more information.


  • VR Geek

    I have read several times now that AR via a pass through camera does not feel right as your eyes suddenly are shifted forward to where the cameras are. I would be curious if HTC solved this issue, but I doubt that very much. This announcement is more like a desparate plee to entice suckers in their overly priced HMD.

    • benz145

      It’s true that the perspective will be incorrect, though the problem is identical for any pass-through AR headset. Camera feed latency is also a problem. However, passthrough AR on a VR headset like the Vive Pro has other advantages (like high resolution, great tracking, and a wide field of view) which today’s opaque AR headsets haven’t achieved.

    • dk

      u can have a software adjustment of the video correcting for that to some extent but it’s not perfect….it’s basically a built in zed mini ….but it’s mainly for developers to experiment with ar …..not really a great working consumer ar product…… also I’m hearing the res and latency is not great

    • Lucidfeuer

      Yup, that was the problem from the beginning, adjusting the perspective is quite (but not so difficult) at least for far away elements, it gets complicated for closer ones like your hands for adjusting the perspective. But as most things, solution already exists thanks to software research, it’s just that they’re not integrated and implemented by corporations…

    • Graham J ⭐️

      Just a thought but if their SDK is capable of realtime reconstruction of the environment then you should be able to move the virtual camera location back a bit.

    • NooYawker

      When I look through the Vive camera everything looks enormous.

  • Harris

    Curious regarding specifications of the box used to drive the spatial scan demo.

  • dk

    it will be cool to check out the specs of the cameras ….res fov fps…added latency
    one dude was saying that from the sdk the specs don’t look good….vga res and 200ms latency

  • dk
  • Badelhas

    What about the normal Vive? Will the camera ever be be used?

    • Laurence Nairne

      Original Vive only has one camera and I don’t think they intended for it to be used in an AR context. It can do passthrough and chaperone type tasks, but nothing to the degree that this video shows on the Pro.

  • Jean-Sebastien Perron

    Yet the Vive Pro (Problems) cannot even proper headphone and microphone.

    • NooYawker

      We get it. The vive killed your puppy. You’re incessant whining about how much you hate the Vive is pathetic.

      • Flamerate1

        After snooping around his profile like the person with good internet etiquette would do, I have confirmed that he is indeed a vive hater.

        Quick question for Jean: Do you have a vr headset and if yes, then which one is it? This is of course to understand your bias if you have one.

        • Jean-Sebastien Perron

          I have a PSVR, Rift and now a Asus MR. My friend has an HTC Vive. I have nothing against Vive itself, but the way the company treat it’s customers is illegal and fraudulent. Vive and Vive pro are the most expensive VR, yet they are at the bottom of the quality in term of graphics and sound quality. Their controller is almost as bad as PSVR.

          • Flamerate1

            Ok, I see. I don’t necessarily see them to be fraudulent, but the price (as well as the initial coverage) is why I bought the oculus.

          • Jean-Sebastien Perron

            It is fraudulent and illegal, they changed the warranty to not cover sweat damage after the product was bought. They ask you to send it for free for repair. They they kidnap your Vive and won’t return it until you pay a 45euros ransom (not repaired). Breaking after 2 months because of sweat is a design flaw and should be covered by warranty because it is normal use of the product. https://youtu.be/ymhQxBalzCE

          • jj

            umm theyre certainly not at the bottom of quality in terms of graphics and sound quality…. Thats just bs and you know it. jean is a troll he uses multiple accounts to bash things he dousnt like and references his friends youtube video as click bait. dont fall for it

          • Jean-Sebastien Perron

            I only have one account and this is my real name, i am not a troll like you. Stop attacking the messenger and try to bring some arguments about the subject.

          • jj

            seriously jean? anybody can look at your post history and realize you’re full of shit. They can also see that you only follow/are followed by one other account who suspiciously has similar posts to your narrative, almost word for word.

            Its funny you try denying being a troll, look at every post you’ve made on RtVR before this one and you’re clearly a troll.

            Don’t come inhere with that bs that you’re not a troll and were just attacking you because you have brought the more harmful vocabulary and posts than anybody on here and there’s over a dozen people reading this right now that will agree with me.

            Some of my posts arent nice but none of them are troll posts like that majority of yours clearly are. Plus you cannot defend any statements you’re outcome is always posting that one video as click bait. I post more on here about how annoying you are than about vr because of how actively you troll.

          • Jistuce

            ” in Canada.”
            Pffft. As if anyone cares about Canada.

          • chris

            thats rubbish the controllers on vive are brilliant what is crap about vive is the none user replacement battery scheme they have reminds me of apple and it´s very frustrating to be treated like an idiot and overcharged for a battery! especially when i want 6 batteries and i´m quite capable of changing them myself

  • Dave

    Does anybody care about the Vive Pro, I mean really?

    • Jistuce

      Yes. It is where I get a good percentage of my laughs lately.

    • jj

      now i do

  • That looks quite good if i’m honest. Certainly good if you want to start understanding and developing AR apps for the future,

    • Graham J ⭐️

      I think that’s the key here – not that it’s necessarily a great AR experience but that it gives you enough to get started developing for an AR future. There aren’t a lot of solutions for that right now.

  • NooYawker

    AR through a camera isn’t the same as AR glasses. But it’s a nice addition. If you have a Vive Pro that’s is.

    • Sponge Bob

      if latency is minimal it can be practically the same – with higher res cameras

      this is not new btw

      See Vrvana Totem headset sold to apple a while ago

      • jj

        maybe in a few years but as of now nothing supports the bandwidth required for these

        • Sponge Bob

          Bandwidth?
          This is all tethered to pc

          • JJ

            yeah bandwidth of usb cables

          • Sponge Bob

            yeah

            I guess that’s why lo-res VGA cameras on Vive Pro

            …and no mixed reality for Microsoft’s “Mixed Reality”

  • Zach Mauch

    This is exactly what I’ve been waiting on!

    • Graham J ⭐️

      Here’s a video where they’re using it:

      youtube . com/watch?v=-gt1vBJqhco

      Quality doesn’t look great there, hopefully it doesn’t do it justice.

      • Jerald Doerr

        Ug!!! I can tell if they’re using crapy video cards or the cams in the vive are slow but that looks like crap…

        • Graham J ⭐️

          It’s the cameras. Saw some other more raw vids posted on /r/vive and it’s all low res.

        • Vive Pro’s cameras are not high res at all

          • Jerald Doerr

            Interesting to see what the max res is for the cameras. I would have put no less than the resolution of the internal screens myself. Question is the max frame per resolution as I’m guessing that would affect the smoothness of the AR.. Ug 30 or 60 fps might look slow with AR.. Something I just don’t know enuff about.

  • Stefan Küppers

    I wish they would go beyond unity and unreal to open this up for webvr/xr.

    • Lucidfeuer

      Aren’t Unity and Unreal already compatible with webvr/xr?

      • jj

        yeah they are but not their vr aspects and plugins.

  • dexkz

    The real question, will it support inside-out tracking?

    • Laurence Nairne

      The real answer: no.

      • Johannes Bühler

        Are you sure about that? Given it allows for spacial mapping it most certainly could be used to significantly improve playspace setup.

        But I know nothing about the latency nor about how persistent that SLAM data is so you’re most probably right that it won’t be possible.

        • dk

          technically it can be done and u can have hand tracking ….so no need for the controllers and light houses …but they haven’t mentioned anything about a software doing that ……and besides it’s just meant for devs to experiment with ar it’s not really a great consumer ar device

        • Laurence Nairne

          As dk says below, it’d be technically possible, but HTC won’t support it because they’ve already got a robust tracking solution and there’s little of commercial value to give an inferior (in terms of tracking accuracy) solution.

          Also, the feed latency seems far too high for use with tracking. I saw someone mention 200ms below!

    • Sponge Bob

      inside-out is inferior to Valve’s laser based tracking, especially in its second generation
      for pro headset tracking has to be 100%

    • Hacker4748

      Well, technically the Vive only has inside-out tracking.

  • RJH

    I’m assuming that this should also be coming for Windows Mixed Reality headsets given what the name implies!

    • Lucidfeuer

      This is indeed what the name means, not just implies…

      • Sponge Bob

        you mean you can actually see through those cameras on Windows MR headsets ?
        I thought they were just for inside-out tracking, no ?

        • JJ

          spone is right they are comepletely different types of cameras. The windows current cameras cannot give you the visable data that normal cameras provide.

          • Sponge Bob

            In other words, Microsoft appropriated (trademarked) the name “mixed reality” without actually providing anything close to it

            Me thinks they should be sued and the name “mixed reality” should be taken away from them (unless they can demonstrate one immediately)

            And they were complaining about “patent trolls”

            how about “trademark trolls” ?

          • JJ

            yes i would love someone to go after Microsoft for that. I feel like the entire first part of thier sales were do to misconceptions and beliefs they they purposefully allowed to happen

  • Lucidfeuer

    So finally they did some of their job somewhat. I will consider the Vive Pro depending on what other upcoming headsets are proposing of the sort.

    • jj

      yeah this really changes things but still needs some time to develop.

  • Nathanael Mooth

    That might almost make the price of the pro worth it.

    • Peter Hansen

      No. The fact that this device also is an AR dev kit increases the price of the product being a consumer headset. The decisions of HTC don’t get any better with this.

  • guest

    If you’ve used the Leap Motion or Hololens, this is not that. The outward cams have a tiny fov. They are slow and nauseating. They provide none of the hand rec, edge detect, or any recognition middleware.. so you have about 50 dev years of catching up to do on that front. We’ve never seen the spatial mapping do anything but crash the unit.

    For commercial applications, we continue to see devs tape better cameras overtop of these cameras because they have no value. HTC has been told this many times and this will confused people on the state of AR which has moved past this. The Pro is great for a lot of things… but this isn’t AR and it can’t be used for playspace set up.

    • JJ

      this is exactly what the Zed mini is like! Tiny fov, slow, nauseating, no tracking and more bad decisions.

      • Keelan_Stuart

        The issue with optical pass-through MR (and in spite of any caveats I’ll mention, it IS the future) is current camera / connectivity / bandwidth limitations. High speed, high resolution cameras require a lot of bandwidth for even a single frame from just one camera… think 2048 * 2048 bytes per frame at 90Hz (that’s 377MB / second!)… so, USB 3.1 data rates are necessary – and camera manufacturers only guarantee performance with specific chipset manufacturers on the PC-side of the connection… and you need multiple controllers, one for each camera. Plus, there are things most people don’t even consider… like the fact that the native pixel format of the sensors isn’t RGB, it’s bayer mosaicked (look it up if you’re curious). Software that relies on the driver to do a bayer-to-RGB conversion will never be fast enough to sync with the 90Hz refresh rate on the headset and will leave you with frame jitter due to mismatched framerates… so, the decoding has to be in a pixel shader. Which leads us to the graphics software itself… PC graphics APIs want the entire texture in memory before presenting the scene to the video display device, meaning you have to wait for a whole frame of video from each camera before rendering – this, opposed to progressively reading and combining pixels as scan lines are captured and output, respectively.
        The point is: once every PC has multi-channel USB 3.1+ connectivity out of the box, this technology can be fully realized and enjoyed by the mainstream… but probably not until then.

        • Keelan_Stuart

          Sigh – I meant to say “The issue with **digital** pass-through…”

          • Sponge Bob

            So its either digital camera pass through with all the issues you mention
            OR
            transparent AR displays like those used by Meta

            for standalone it’s most certainly transparent display technology
            although it wont allow all the tricks afforded by fully digital
            like placing real objects in foreground or background

        • jj

          thats very insightful thank you!

  • Jerald Doerr

    Saw this coming a long time ago… I see they basically took the safe route with the vive.. Seems like they could combined camera tracking with the lighthouses tracking to improve tracking even more. Guess will have to wait for that but I’m disappointed to hear the cams are basically crap.

    • Sponge Bob

      lighthouse tracking can’t be improved by adding inferior inside-out tracking

      • Jerald Doerr

        Only because you say so??? I’m 100% sure you can combine deferent methods of tracking to assist when one fails. They already do this in defern’t existing applications.

        • Sponge Bob

          adding additional base stations for Valve tracking (Generation 2) will give much better results than adding inferior inside-out tracking in case of possible occlusions or for extending the range

          • Jerald Doerr

            Will Gen 2 let you do that without slowing down the rate like people are suggesting gen 1 would!?

          • Sponge Bob

            the IR flash is removed from Gen2 so I assume if multiple base stations can be synchronized to each other (e.g. via sync cables) you can have many of them in the same area without slowing down update rate
            it’s a cumbersome installation indeed but much superior to inside-out tracking (which relies on environment too – try it in a big hall with white ceiling and walls – it simply will not work)

  • Cat of Many Faces

    looks promising!

  • sfmike

    Love the special scan!

  • Peter Hansen

    So basically, the Vive Pro is not a finished product, but a developer kit. That is why it is neither targeted at consumers nor the business sector. Now it makes sense.

  • Amazing

  • dk