The developers behind Modbox (2016), the VR physics sandbox for SteamVR-compatible headsets, have been experimenting with augmented reality recently, something they say is a perfect fit for the Modbox software.

Developer Lee Vermeulen didn’t wait for VR headsets like the Vive Pro to get an AR function, but rather strapped a stereoscopic Zed Mini to his original HTC Vive in effort to push Modbox forward into augmented reality.

“We are planning to make Modbox into a AR building application, which we feel it’s perfect for,” Vermeulen told Road to VR.

Image courtesy Stereolabs

In VR, Moxbox basically offers users a sandbox to experiment with toys, mods, and a place for impressive object destruction, although the app has taken a slightly more serious tone lately with its recent update which allows multiple users to concurrently script entity behavior much like you can in the Unity game engine. Modbox is currently in Early Access on Steam, supporting HTC Vive and Oculus Rift, although that may change in the near future.

AR games, Vermeulen tells us, “need to match the environment they’re in; so to do that, games usually tend to adjust based on the environment in a procedural way. For Modbox though, we’re creating something that allows people to make games to exactly match their environment, and hook up their AR worlds to their smart home.”

Modbox AR Experiments

While the team hasn’t discussed when they’re releasing the AR version of Modbox, Vermeulen highlighted a few interesting use-cases, including a way to change the color of his WiFi-connected Philips Hue Lights, and to turn each light off by ‘shooting’ it with a virtual bow and arrow:

Meta's Former Head of VR: Oculus Go Was His "biggest product failure" & Why it Matters for Vision Pro

Here, they show off what a self-made AR version of Space Pirate Trainer (2017) might look like. Drone battle in the living room anyone?:

These AR experiments are possible because Vermeulen previously mapped his environment and placed the virtual items in himself, something that requires a little DIY, but entirely possible thanks to the application’s numerous entities including weapons, objects, and NPCs.

Below we see Vermeulen importing a NPC and playing catch with a virtual basketball. To create the effect of ‘pulling’ the NPC from his monitor, he set a virtual camera to match the monitor’s position, making it seem like a magic window.

Virtual reality developers eyeballing augmented reality can get started using this setup, although admittedly the ZED Mini camera + VR headset combo actually overshoots the mark somewhat. It relies on a gaming computer’s elevated graphical capabilities, and presents the user with a comparatively wide field of view not possible on current AR headsets. The Vive controllers, which have 6 degrees of freedom (6DoF) positional tracking, are also unique to VR at the moment, although Magic Leap One is said to ship later this year with a single 6DoF controller which could allow for many of these sorts of interactions.

Just like in the early days of virtual reality, the first developers to create something fun and useful are kind of like the first miners to a gold rush. So while AR headsets still have a ways to go, seeing these early stabs at solving the question of content gives us an exciting look into what could be the near future of AR head-mounted displays.

Newsletter graphic

This article may contain affiliate links. If you click an affiliate link and buy a product we may receive a small commission which helps support the publication. More information.

Well before the first modern XR products hit the market, Scott recognized the potential of the technology and set out to understand and document its growth. He has been professionally reporting on the space for nearly a decade as Editor at Road to VR, authoring more than 3,500 articles on the topic. Scott brings that seasoned insight to his reporting from major industry events across the globe.
  • jj

    yeah this video footage doesn’t justify how bad the zed works and looks through the headset. First off Stereo labs lied on thier website about the FOV of their device and only changed it after we contacted them about our disappointment in their product due to misinformation they had been spreading. So be careful with the Zed mini because we returned ours and don’t plan on working with stereo any more.

    • JJ

      Not to mention that this captured footage looks good quality but the Zed only shows the user 720 at 60 fps or 1080 and 30 frames which is not fast enough for vr use and this is the players main view of the world while inside the headset. So idk you tell me if you think you can use vr at 30 fps….Again we were really excited for the Zed and were one of the first buyers of the mini, but upon receiving it we were so badly disappointing by some of the details that we felt stereolabs purposefully left out or straight up lied about. They should be more honest with their products otherwise they can expect more people to feel passionate enough to write a review like this.

    • Lucidfeuer

      Have you tried one, is this based on lots of review or just a few? Given lazy/greedy manufacturers have belated, even vaporwared the obligatory integration or AR tracking into headsets (as far as the Oculus Crescent Bay), we’re starting from scratch and this is probably the most challenging R&D front for virtual headset. That’s why it’s not so soon that there are finally solutions which can somewhat do pass-through AR, but it’s way more challenging than it looks.

      • jj

        yes we bought one for our office where we develop lots of ar/vr applications. Some of our work made it to CES Vegas at the start of the year :).

        So we have a lot of experience with new hardware and I specifically program for every device from cardboard to the hololens and soon the magic leap.

        Bottom line this zed is pretty crappy and I question wether its a scam or not after seeing the product. The videos in the post are captured off of the pancake display. Because what you see in the headset is not nearly that nice of quality and your fov is around half of what you have in the vive and the rest is just blackness. There is no sense of reality more like a flat panel inside vr that has a camera feed to it.

        We were really really excited for this modular AR solution and i think thats why I’m so passionate about how bad it turned out. Not to mention they said to us that a lot of other devs are complaining to them in emails about these issues.

        oh and funny thing they have usb-c on the device BUT the usb c only works when orientated one specific way, which completely disregards one of the biggest improvements usb c provides. This also provides a headache because nobodys going to think to revers the usb c plug and the device will throw up errors if you have it wrong that wont tell you that. Plus they provide two usb cords that other devs and I on forums have confirmed that the longer cord is too long for the device to work on. It cant send the data fast enough that far i guess because the zed requires a ton of bandwidth. Again this has been confirmed by others as well as me, so a lot of us are questioning their decisions.

        So a lot of issues right out of the box. I really wanted it to work and was afraid it was us making mistakes but after a couple of weeks and many headaches we threw in the flag because all other AR developing we’ve done has not been nearly that janky.

        • Lee Vermeulen

          Hey, I wanted to respond to this, since I definitely love the Zed and what that company is doing (I am the dev mentioned in this article)

          I am not sure what you expected FOV wise – or in headset experience, but Zed definitely advertised exactly what it was.

          You should have done your research before buying – rather than call it a ‘crappy product’ because its not what you naively expected.

          Having had experience with early VR/AR hardware, I can say it’s been a smooth process. There were technical issues to overcome to make it work with my software but thats expected with new hardware like this. It sounds like this sort of hardware is not for you – it’s not meant as a consumer AR headset product, it’s meant for devs to work with and experiment with.

          • Jerald Doerr

            Lee…. Little jj talks like he knows everything but he really knows how to troll..

          • jj

            well we actually caught them red handed lying about the FOV on their website and after confronting them about it they agreed the website was claiming a bigger fov and changed it. Good on them to fix it but bad on them for “accidentally” messing it up to start with. It clearly is a developer device but not very practical for whats its trying to do.

            Thank you for you feedback. Your work as fantastic btw, I’ve been following your for a while now to see whats possible and what isn’t because you make things so new and fast. Please keep developing

    • Graham J ⭐️

      Given the new Vive Pro SDK that just released I guess he’ll be switching to that anyway.

      • JJ

        right im so excited for that!

  • Very cool experiment. And now that the Vive Pro has AR feature, they can integrate with it

    • JJ

      Its funny stereolabs spent so much time developing something that the vive just added on a whim. And stereolabs sdk is only going to be better for a short time because of all the resources HTC has to work with.