Valve’s Index headset has had a passthrough view feature since launch, but the company has just released an experimental update which aims to make the view through the cameras more realistic by creating a synthetic image that more closely matches what your eyes should actually see.

A week ago, if you put on a Valve Index headset and double-tapped the menu button, you’d be greeted with a ‘passthrough’ view of your surroundings. Although this view is technically stereoscopic, it isn’t showing you exactly what you’d see if you weren’t wearing the headset, largely because the cameras on the headset are not in the exact same position as your eyeballs. In fact, the cameras are several inches away from where your eyes actually are, and they’re spaced much further apart.

How to Opt into SteamVR Beta

In your Steam games list, right-click on SteamVR > Properties > Select the ‘Betas’ tab

In the drop down list, select ‘SteamVR Beta Update’

Allow SteamVR to update

This week Valve released and experimental upgrade to Index’s passthrough view that’s designed to fix these problems and give you a ‘stereo-correct’ view. Valve calls it ‘Room View 3D’, and if you have an Index headset yourself you can check it out today by installing the SteamVR beta version 1.13, and then enabling ‘3D’ in SteamVR’s ‘Camera’ options. In this update, Valve also updated Index’s firmware to reduce the latency of the headset’s cameras.

Captured by Road to VR

As for Room View 3D our understanding is that, rather than simply showing you exactly what the cameras are seeing, the view from each camera is being compared to assess the depth of each part of the scene. The scene is then ‘distorted’ from the raw camera input to more closely match what a pair of cameras would see if they were in the exact same position as your eyes.

SEE ALSO
Snapchat Reveals Latest Spectacles AR Glasses with Attractive Pricing for Developers

We tested out Room View 3D and could see the benefits of the stereo-correct view. When the system was working correctly, everything felt a bit more realistic, especially the size of your hand in front of the headset.

Image courtesy Valve

However, the system is highly unstable at this experimental stage; even just looking slowly around the environment would show depth artifacts which look like the parts of the scene are glitching. Occasionally when bringing my hand close to the headset it would be stretched and distorted across several different depths. I would guess that the low light sensitivity of Index’s cameras are a contributing factor.

Captured by Road to VR

Valve’s Room View 3D approach is similar to the ‘Passthrough+’ feature on Quest and Rift S, though Oculus’ solution is much more stable and usable at this point, even in similar lighting conditions. Valve’s “experimental” approach will ostensibly improve over time; the company notes that “this functionality is rapidly evolving, so we want to hear your thoughts and feedback.” Valve says it worked with Arcturus Industries and Occipital to create the underlying computer vision technique.

Ahead of the headset’s launch, Valve talked up the cameras on Index as being ideal for computer vision, but nearly a year after launch the company had done very little with them beyond the initial passthrough view—until now. Unfortunately the original passthrough view and the new Room View 3D continues to suffer from a long (several second) activation delay, making the feature much less useful for quick peaks outside of the headset. By contrast, activation of passthrough on Oculus’ headsets is nearly instant, making it more viable for quick looks to talk to someone or to make sure nothing has entered your playspace.

SEE ALSO
VR MMO 'Zenith' Releases Final Content Update & Drops Price Amid Development Halt

Valve says that users can “expect to see more experiments over the coming months,” and there’s hope that the work could eventually result in the passthrough view being used in SteamVR’s boundary setup process, as Oculus’ latest headsets have proven that this makes for a more fluid setup experience.

Newsletter graphic

This article may contain affiliate links. If you click an affiliate link and buy a product we may receive a small commission which helps support the publication. More information.


Ben is the world's most senior professional analyst solely dedicated to the XR industry, having founded Road to VR in 2011—a year before the Oculus Kickstarter sparked a resurgence that led to the modern XR landscape. He has authored more than 3,000 articles chronicling the evolution of the XR industry over more than a decade. With that unique perspective, Ben has been consistently recognized as one of the most influential voices in XR, giving keynotes and joining panel and podcast discussions at key industry events. He is a self-described "journalist and analyst, not evangelist."
  • Xron

    Interesting…

  • jackie mclonghands there

  • impurekind

    Well this is kinda embarrassing for Valve considering how good it already is on Quest.

    • Rosko

      Yes cameras are still a bit of a gimmick on the index.

  • care package

    I don’t get why the passthrough view has to suck on all the HMDs.

    • Adrian Meredith

      Because it’s incredibly difficult to achieve something that looks natural when the cameras aren’t in the danger position as your eyes. This is even more difficult on the quest as there are no front facing cameras

      • Alexander Grobe

        Apparrently Doc-Ok published a solution to that problem years ago but no HMD manufacturer has picked this up yet.
        https://uploads.disquscdn.com/images/a13df8f6408a8fc90591516a5e4ed4989dc8145330e5a4871f11200ab1c056c6.jpg

        • Rosko

          Maybe i’m being obtuse here but why would you need to refract the image with mirrors to the eyes? The cameras are digital. The issue is surely to do with their position in the first place & the fact that they see 3d objects at a different angle. That’s what the reconstruction is doing i assume.

          • Lachlan

            Well because if the cameras are positioned behind the 45° mirror the same distance your eyes are, then what the cameras ‘see’ is identical to what your eye would ‘see’ (at least, in regards to positioning)

            If you were looking at yourself in a two-way mirror that was 1m away from you, you’d be seeing an identical thing to someone standing 1m behind the two way mirror looking out at you. Obviously, the image would be flipped by the mirror, but it’s trivial to un-flip the image the cameras would see.

        • Karol Gasiński

          I would say this diagram has nothing to do with pass-through issue described in article. In diagram we see half reflective mirrors, 50% of light goes straight-through to eye, and 50% is reflected to camera. Camera is seeing exactly the same thing that user is seeing, but if in front of this diagram would be typical display of VR headset, whole thing wouldn’t make sense because you could just use image displayed by panels directly (you already have them in memory after all), so why would you re-register them with cameras again? This diagram makes sense only if it’s for see-through system, where user is seeing physical world, and designer of the system would want cameras to register exactly the same image from the same perspective. For this to work with AR glasses, transparant displays would need to be placed between mirrors and eyes. Yet it still doesn’t make sense from use case point of view. To have SLAM/VIO tracking you don’t need to see exactly the same thing as user does (perfect example here is Oculus Quest with cameras being placed in completely different places). Thus, to conclude, this diagram for me makes sense only in one case: if person wearing this is supposed to be “live camera”, recording everything that user sees in exactly the same way, so that this recording later can be displayed in the VR headset (or streamed to it). So completely different use case.

        • psuedonymous

          That’s the decades-old Camera/Aremac setup (see: Steve Mann’s various generations of EyeTap devices). It has significant limitations on possible field of view without using enormous mirrors.

        • Mradr

          Wouldn’t that mean Doc-Ok would hold rights thus unless a company wants to spend a lot money to use said design and increase HMD cost – why would they?

          • Alexander Grobe

            Doc-Ok posted his idea on his blog. I’m not a copyright or patent attorney, so I can’t asses how much this idea is protected or not.

  • Michael Geoffrey

    There is an app on steam called FragmentVR where the dev allowed users to cut out a view in VR to see your surroundings (like for a button box for flight/race sims), and it worked on the OG Vive and then he planned to make it work for Vive Pro and just gave up. It was such a great idea, but i havent found another app like it for any other headset like the Vive Pro or Index yet.

    • KydDynoMyte

      Have you looked at Reality Blender? Perfect for this.

      • Michael Geoffrey

        Really! Wow i saw it before but didnt realize they got it to work.

  • Jetson

    Yeah great, fantastic, awesome, magnificent, can’t wait to test it… IF ONLY THAT VALVE INDEX ORDER I MADE ON MARCH 9TH WOULD EVER ARRIVE… maybe this year!

    • metanurb

      Did you get it last year? :P

  • metanurb

    The Quest passthrough is pretty distorted also, especially at closer range. But it’s not too bad, it’s very useable to navigate your real environment. It’s only black/white though. But a good “bonus” usage of the inside-out tracking system. And the tracking is pretty damn good I have to say.

    The hand tracking thing is more of an experimental gimmick at this time though.

    • silvaring

      Hasn’t it been like four years since Leap Motion hand tracking demos already started being released for the DK2? What the heck… its taking ages.

        • Private draken

          The Leap is a lot better than Oculus Quest Hand Tracking. Its more responsive, and can capture fingers while the hand is pretty much closed.

          Its a little heater through, any HMD you hook it up to will increase in temps very fast. Also uses a decent amount of compute power(compute is done on PC, not device).

          • Thanks for the information, I’ve been tempted to get an ultraleap for my Index.

          • silvaring

            What is it about hand tracking that is so much harder than inside out head tracking (like WMR). Is it the fact you need to predict individual finger movements instead of just a neck / head tilt, and that increases the complexity substantially? I don’t get it.

          • Private draken

            Its just way more things to track, with less data. For the headset tracking, the headsets find all sorts of stuff around the room to ‘ground’ itself. All of that is used to track the single ‘object’ that is your head.

            When it comes to hands or controllers, things are different. The leap motion projects IR dots all over your hand, then use cameras to track those dots. Size of dot can tell them distance from camera etc, lots of extra data.

            Quest, is trying to take just pure black and white cameras feeds in stereo, to track the hand, hence why its not as good, at least not yet. Machine Learning is pushing every single tech/field/idea beyond what they ever thought was possible. Not just in VR, in every aspect of life.

  • I expected those cameras to be exploited more, given my love for passthrough AR… but it seems that Valve is taking them as a side project. It’s weird they put them in that position if they wanted to use it for AR though… whey are not they in front of the eyes like with the Vive Pro?

    • Here’s an idea for HMD manufacturers, fix external camera to match lens centre of each display/lens block, the camera moves with the lens.

      Adjusting IPD then sets the external camera view to match. Simpler passthrough AR with less compute required for correction?

  • Private draken

    Eh.
    SteamVR has the same kind of passthrough that Oculus had, years ago. You even reported on it back then lol ?

    Why is the new idea floating around now, that SteamVR never had passthrough before, and now this is it ?

    It isnt. SteamVR had passthrough before the Rift S or Quest launched.

    This is simply a different ‘mode’ for it, which is closer to a proper reconstruction, compared to what Oculus or the original SteamVR passthrough does.

  • Clem McAree

    Wonder if this feature will become available on the upcoming HP Reverb G2?