Varjo’s XR-3 headset has perhaps the best passthrough view of any MR headset on the market thanks to color cameras that offer a fairly high resolution and a wide field-of-view. But rather than just using the passthrough view for AR (bringing virtual objects into the real world) Varjo has developed a new tool to do the reverse (bringing real objects into the virtual world).

At AWE 2021 this week I got my first glimpse at ‘Varjo Lab Tools’, a soon-to-be released software suite that will work with the company’s XR-3 mixed reality headset. The tool allows users to trace arbitrary shapes that then become windows into the real world, while the rest of the view remains virtual.

This makes it possible to bring parts of the real world into virtual reality. For instance, as the company shows, you can trace the outline of a physical steering wheel peripheral so that you can see the real wheel inside of the virtual world.

The Varjo Lab Tools software supports three core masking modes: static, depth, and marker-based. With the static mode you can draw a passthrough window that will essentially be tracked to your head. With the depth mode you can have the system automatically pull in anything that’s within a certain distant to the headset (judged by the lidar sensor). You can imagine this being used to automatically show your real hands holding something when you raise them up in front of you, without pulling in the background of your room.

In the marker-based mode you can create a window that’s tracked to a simple fiducial marker and moves with the marker. This would be great for something like the steering wheel example above, or allowing your real keyboard to persistently appear in the virtual world. The markers can be easily printed and mounted anywhere… and you can track up to 1,000 of them.

At AWE 2021 I got to try the system for myself, and I was impressed with the user-friendliness of the tool. To draw a marker-tracked mask, for instance, you just use a VR controller to select the marker you want to track, and then trace out the mask with a simple point-and-click system. For example, I used the tool to mask out an area around a gamepad that had one of the markers attached to it. Then when I jumped into VR I could see a window around the gamepad, allowing me to look down and see my real hands on the gamepad and easily reference the position of my fingers with regard to the buttons.

And what’s particularly cool is that all of this passthrough magic is happening completely independent of the VR application, which means the passthrough windows that you draw can function inside of any app without a special integration by the developer.

Apple’s Official Vision Pro Case May Be Pricey, But It Saved a Headset From a 4,000 Pound SUV

It’s a very cool system that makes passthrough fundamentally more useful, but it isn’t yet flawless. One of the most obvious limitations right now is the update rate of the masked area. While the view through the passthrough window (ie: the view of the real world) updates at a perfectly fast rate, the shape of the window itself (when attached to a marker) only seems to update a few times per second. So if you’re moving the marker even slightly fast you’ll see the passthrough window lag behind as it tries to keep up.

I’m not sure what the limiting factor is on the update rate for the passthrough window, but I take it this is something that will be improved in the future. A much faster update rate would make real-world objects appear much more seamlessly within the virtual world.

– – — – –

Varjo says that the passthrough masking capabilities can also be used with the chroma key functionality it has introduced previously, making the XR-3’s passthrough system very functional and flexible compared to anything else on the market today. Indeed, just as the impressive resolution of Varjo’s flagship headsets has long felt like a glimpse of the eventual future of consumer VR headsets, we’d also hope to see this kind of advanced passthrough functionality come to consumer headsets, eventually.

For now the capability is only on Varjo’s high-end enterprise XR-3 headset (as it’s the only one with all the passthrough hardware necessary to make it all happen). The company says the Varjo Lab Tools software, which will enable these features, will launch alongside the next major update to its core ‘Varjo Base’ software.

Newsletter graphic

This article may contain affiliate links. If you click an affiliate link and buy a product we may receive a small commission which helps support the publication. More information.

Ben is the world's most senior professional analyst solely dedicated to the XR industry, having founded Road to VR in 2011—a year before the Oculus Kickstarter sparked a resurgence that led to the modern XR landscape. He has authored more than 3,000 articles chronicling the evolution of the XR industry over more than a decade. With that unique perspective, Ben has been consistently recognized as one of the most influential voices in XR, giving keynotes and joining panel and podcast discussions at key industry events. He is a self-described "journalist and analyst, not evangelist."
  • Holdup

    Now I can track my beer while playing poker star vr

  • Ad

    There was something like this on Steam, too bad it didn’t catch on:

    • zmagzem

      I even have got paid $19,350 in less than five weeks simply working easy jobs at home.~q12~As I’d lost my previous post, I was so perturbed and thank God I have started this super on-line assignment that’s why I can figure out how to possess thousands from home.~q12089~Any person certainly can try this best career and may collect extra dollars online viewing following web.

      >>>> >>>> >>>>….09


  • Martijn Valk

    A bit rough around the edges but still, impressive stuff and I definitily can see the advantages with this!

  • psuedonymous

    Ah, Augmented Virtuality: the 4th corner of the real/virtual worldspace that rarely gets a mention, unlike Virtual Reality and Augmented Reality.

    • Yes, finally!

      • anselmawqol

        I’ve even produced $17710 just in 21 days simply working a few easy tasks from my apple laptop.~o136~Since I had lost my last business, I was unsettled and thank God I got this top offer so this way I’m capable to achieve thousand USD directly from my home.~o1327~Anyone can try this offer and may make more dollars on-line viewing this web-page.

        >>>> >>>> >>>>….27


  • Very cool. I’m envious because at AWE they didn’t let me try it!

  • Keng Yuan Chang

    So, that’s metaverse right there, just need to shrink the size and price by 90%.

  • Christian Schildwaechter

    Really useful feature. While it is certainly most useful with hires color cameras and extra LIDAR information for depth dependent passthrough, the marker based version could be implemented for pretty much any HMD that uses inside-out tracking. The Quest cameras have sufficient resolution, and I don’t mind pasting large fiducial markers onto my keyboard, thermos, tea cup or bowl with snacks, nor having to grab them in paranormal grayscale night vision mode.

    Recognizing markers is computationally much cheaper than hand recognition, so maybe Meta or Qualcomm/Pico will offer this option in a future SDK. Unfortunately VR app developers cannot implement this themselves, as they are not allowed to access the raw camera feed for privacy reasons.

  • Arno van Wingerde

    But to an extent, Quest stated doing this as well, for instance putting your coach or table in.

  • johann jensson

    I always thought that people use VR to escape the real world. Weird!