Thanks to the tracking cameras on Rift S, the headset also comes with a pass-through video feature which Oculus calls ‘Passthrough+’ (with the ‘+’ denoting its low latency, stereo-correct, wide field of view). While it’s the best pass-through video we’ve seen on any consumer headset so far, the feature will soon be enhanced to take advantage of new capabilities available in NVIDIA’s latest GPUs. Oculus’ motion smoothing tech, Asynchronous Spacewarp, will also be similarly upgraded.

While Quest also has a passthrough video function, Oculus didn’t give it the ‘Passthrough+‘ designation because of its latency and lack of stereo depth. On the Rift S however, the company was able to leverage the power of the PC to computationally solve for the depth of the scene around the user, and then reproject the view accordingly to provide stereo depth. Along with lower latency, this makes Passthrough+ on Rift S look and feel more natural.

On the Oculus Developer Blog the company today explained that Passthrough+ leverages technology the company originally built for Asynchronous Spacewarp (ASW), which keeps motion smooth inside the headset even when framerate is unstable. ASW does this by comparing the motion of previous frames to estimate how that motion should continue into the future (and from there synthesizes a new frame based on the estimation). For Passthrough+, similar tech is used to compare simultaneous frames from the Rift S cameras to understand the distance between objects in each frame, and then infer the depth of the scene (similar to how we understand depth in the real world by perceiving it with two eyes, each with a slightly different perspective).

SEE ALSO
Oculus Launches ASW 2.0 with Positional Timewarp to Reduce Latency, Improve Performance

The method is similar to the what’s used by most video encoding tech, and ASW and Passthrough+ today use the video encoder on NVIDIA and AMD GPUs to do this work ‘asynchronously’ (without impacting the work the GPU needs to do to render the VR world in the first place).

But an upgrade for Passthrough+ and ASW is on the way and will take advantage of capabilities in NVIDIA RTX (and other NVIDIA ‘Turing’ GPUs), which support new optical flow technology from NVIDIA. [Update: Oculus has confirmed the ASW upgrades will also come to original Rift]

NVIDIA optical flow quadruples the macroblock resolution, increases motion vector resolution, enables following objects through intensity changes, and emphasizes plausible optical flow over compression ratios. The result is half the average end point error than traditional video encoding motion vectors. The qualitative results are equally impressive. With ASW, near-field objects track more reliably. Swinging flashlights hallucinate motion much less frequently and the increased precision means movement is tracked more accurately to individual particles and objects.

The end result, Oculus says, is more accurate motion estimation which means fewer artifacts when ASW is in use.

This illustration shows the increase in motion estimation density and precision in the optical flow method

Similarly, Passthrough+ on Turing GPUs will make use of optical flow upgrades to improve the depth and stability of the view.

For Passthrough+ this means increased stereo resolution of the projected world, while thin objects are correctly tracked and followed. When faced with low contrast or over-exposed areas, NVIDIA optical flow can still infer meaningful disparity values, preventing visual holes or missing data as we estimate the depth of the scene.

This comparison between the old method (left) and new (right) shows how much more stable and accurate the optical flow method is for determining the depth of the scene. (the computation depth reconstruction is inset in each view, enlarge and toggle HD for better visibility) | Courtesy Oculus

Oculus says that the enhancements will launch in June, and be compatible with Turing GPUs running the latest drivers.

Beyond that, the company sounds bullish on what other doors NVIDIA’s optical flow tech may have opened.

This isn’t the end of the line for ASW and optical flow. As demonstrated here, optical flow doesn’t just track movement, but also informs how scenes are arranged spatially. There’s a great deal of information we can glean about environments using optical flow, none of which we can do well with a traditional video encoder approach. The optical-flow SDK opens up many opportunities and research, so be sure to check back for more learnings + insights!

Newsletter graphic

This article may contain affiliate links. If you click an affiliate link and buy a product we may receive a small commission which helps support the publication. More information.


Ben is the world's most senior professional analyst solely dedicated to the XR industry, having founded Road to VR in 2011—a year before the Oculus Kickstarter sparked a resurgence that led to the modern XR landscape. He has authored more than 3,000 articles chronicling the evolution of the XR industry over more than a decade. With that unique perspective, Ben has been consistently recognized as one of the most influential voices in XR, giving keynotes and joining panel and podcast discussions at key industry events. He is a self-described "journalist and analyst, not evangelist."
  • Firestorm185

    Wow, that sounds awesome! Another reason to get an RTX eventually, I suppose! XD

    • Ted Joseph

      I considered that, but I have a GTX 1080 so I am going to stay with that for now.

    • Andrew Jakobs

      OR an 1660?

  • Ted Joseph

    Damn you Oculus!! I purchased the Quest day one, and then folded, and purchased an S on Amazon tonight…. Long shifts ahead to pay off this debt! Oh well, I am sold on Oculus products! Had the Rift since day one, purchased PSVR (aim controller is cool), and now purchased a Quest (love it), and purchased an S for the awesome games coming out soon for the higher end PC gaming! Life is great!!!

  • Niklas Fritzell

    Does not seem loke this ia a feature requiring an rtx card. Any card with turing will do, so this must mean that Nvidia is deliberately ditching Pascal cards?

    • Adderstone VR

      “any card with turing will do”
      No Pascal cards have Turing, Turing is a microarchitecture developed by Nvidia as the successor to Volta which is the successor to Pascal.
      So, yes if you need Turing cores then you need an RTX card or 1660 range card, but Pascal won’t do.

  • s.j.adamson@gmail.com

    this upgrade for asw is only going to be available to rift s users and not rift cv1 users?

    if that’s the case, it looks like my upgrade path is confirmed and ill be sailing off the oculus boat.

    It would be incredibly cheep of oculus/facebook to make a pc side software change/upgrade that has nothing to do with the actual headsets capabilities and lock it down to only the latest headset.

    • benz145

      Good question. I’ll double check.

      • Passthrough or Passtrough+ will NOT be available to CV1 for obvious reasons (I doubt they can extract 3d room information from the external CV1 sensors in a useful way).
        However the improved motion tracking system will most probably be available for all headsets, as long as they use a GPU that supports this new feature. I see no reason to artificially add maintenance overhead of a different feature branch for CV1 just for sake of messing with existing users, specially since they are fully aware the hardcore enthusiasts are probably staying with CV1 and not upgrading to S.

    • Jens

      So they should just stop innovating in the case those that purchased older products get mad they dont get new features? What kind of idiot argument is that? Really this Oculus bashing has gone way to far now, nothing they do can be right apparently.

    • davedsone

      Actually, since it is a function of cameras the CV1 DOESN’T EVEN HAVE, it shouldn’t impact you at all. They are talking about the upgrade in terms of passthrough+, not anything else. But feel free to move on since you apparently are the Prince of Butthurt.

      • CursingLlama

        Actually, the Asynchronous Spacewarp (ASW) benefits have nothing to do with the cameras on the Rift S. If you read the article it talks about upgrades to both passthrough+ and ASW using the RTX cards. So theoretically CV1 could benefit off of the upgrade to ASW.

        Honestly though if your headset is using ASW on an RTX card there is probably something horribly wrong.

  • Luke

    will work on nvidia 1070?

    • johann jensson

      Didn’t you read the article?

    • Nejham Mosquera

      Nop, that’s based on Pascal architecture, not Turing.

  • Kellin Bershinsky

    It seems like the Passthrough+ could also make use of the RTX’s denoising capability thereby removing the graininess of each image. I’ve also seen machine learning algorithms that can add color to black and white images. The integration of the two ML models could overcome the limitations of the Rift S cameras and if the latency of the denoising and color conversion can be kept low enough, the Rift S could become the first quality AR passthrough headset!

  • Amazing! So bad that I don’t have an RTX card…

    Anyway, I think that the update on audio is what the Rift S truly needs to have now