Update: ZED Mini Turns Rift and Vive into an AR Headset From the Future

Shipments starting on December 18th

26

Stereo camera company Stereolabs has launched pre-orders for the ZED Mini, a smaller version of their stereo depth-mapping camera which fits on a mount made to attach to VR headsets like the Rift and Vive. When attached, the camera provides stereo pass-through video and real-time depth and environment mapping, turning the headsets into dev kits emulating the capabilities of high-end AR headsets of the future. The ZED Mini will launch in November.

Update (12/12/17): Stereolabs began ZED Mini pre-orders in September and now the company says they plan to begin their first shipments on December 18th. The company also confirms that each ZED mini will come with a VR mount for either the Rift or the Vive, and 720p video at 60 FPS for the AR mode (with lower FPS options up to 2K resolution).

Stereolabs also intends to update their ZED SDK to includes plugins for Unity and Unreal that “make it easy to integrate virtual objects, lighting, particles, and effects that properly integrate and collide with the real world. The SDK will come with several example apps. Further, the company says that developers will also be able to integrate Vive, Touch, PS Move, and other controllers into pass-through AR experiences.

Original Article (9/26/17): Apple’s ARKit and Google’s ARCore may be bringing AR tracking capabilities to smartphones, but it will be years yet before high-end immersive augmented reality headsets hit the consumer market. Today’s AR headsets, like the HoloLens and ODG R8, have very small fields of view compared to today’s VR headsets (~40 degrees compared to ~100).

The ZED Mini camera, attached to a VR headset, effectively emulates the sort of experience that AR glasses will hopefully achieve in the future—an immersive, wide field of view with real-time tracking and environment mapping.

With a special mount designed to attach to the Rift or Vive, the ZED Mini has two cameras spaced at 65mm (close to the human IPD average), allowing for comfortable pass-through stereo video into the VR headset. In addition to pass-through video, the camera compares the two different images from each camera to build a depth-map of the scene. The company claims the camera can detect depth out to an impressive 15 meters (49 feet). The camera also builds a geometric map of the environment in real time and fuses the data with an onboard IMU enabling positional tracking of the headset within the AR environment.

I got a chance to try the ZED Mini attached to a Rift for myself at VRDC Fall 2017 last week and came away quite impressed. Although the camera’s own field of view isn’t quite wide enough to completely fill the Rift’s field of view, it still presented a large, immersive view of the real world in front of me, far surpassing today’s AR headsets.

SEE ALSO
Lucy Liu Stars in VR Adventure 'The Pirate Queen', Now Available on Quest & SteamVR

At first I was just looking at the crowd standing in front of me. Then I saw a floating menu representing several different demo experiences I could launch. I launched the droid attack game which began to spawn spherical, floating droids in the world around me. As they approached, I saw convincing occlusion as the drones flew behind people and objects nearby. That meant I could duck behind the table in front of me and the drone would be hidden from my field of view by the table, as if the drone was really behind it. As I moved my head around to get a feel for the occlusion, the inside-out positional tracking held up fairly well in my time testing the device, though I’d want a more extensive testing session to get a better feel of the tracking and pass-through latency; I was quite distracted by the impressive occlusion.

In another demo I held a lightsaber prop which was sensed by the camera and a properly glowing end was overlaid on top of it. Using the prop I was able to bat the drones and send them flying to an explosive death. A strong swing would send them flying fast while a gentle poke would be reflected as a mere shove; a rather convincing fusion of an arbitrary, unmarked prop into an AR scene.

There’s pros and cons to doing pass-through AR instead of transparent AR. For one, the augmented parts of the world can look quite a bit more real because the pixels are drawn directly on top of the image of the real world, eliminating that semi-transparent ‘hologram’ look that you’ll find on transparent AR displays (caused by the fact that it’s difficult to create pixels that can occlude 100% of the light from the outside world when dealing with a transparent display). On the flip side, our eyes have excellent resolution and contrast ratio, which generally means we limit our dynamic range (the ability to see dark and light areas of a scene at the same time) and visual fidelity when using pass-through AR (not to mention the potential to introduce latency).

Either way, if devs want to get a head start on AR development for high field of view AR headsets of the future, the ZED Mini seems like it warrants serious consideration.

Image courtesy Stereolabs

Last week the company launched pre-orders for the device priced at $450, with plans to begin shipping in November. As an add-on to the Rift or Vive, it’ll appeal moreso to developers who already own those headsets, while developers just getting starting in the world of immersive computing might also consider the Meta 2.

This article may contain affiliate links. If you click an affiliate link and buy a product we may receive a small commission which helps support the publication. See here for more information.


Ben is the world's most senior professional analyst solely dedicated to the XR industry, having founded Road to VR in 2011—a year before the Oculus Kickstarter sparked a resurgence that led to the modern XR landscape. He has authored more than 3,000 articles chronicling the evolution of the XR industry over more than a decade. With that unique perspective, Ben has been consistently recognized as one of the most influential voices in XR, giving keynotes and joining panel and podcast discussions at key industry events. He is a self-described "journalist and analyst, not evangelist."
  • That’s a very interesting project! The problem is that $450 is really expensive… and we don’t know what are the complete capabilities of this device.

    • Get Schwifty!

      Jeeze man, you want it all for nothing. Considering its the price of a modest phone these days $450 is not all that bad for ‘new” tech.

    • chuan_l

      Presumably its a ” mini ” ZED depth camera —
      All the spatial mapping you can eat , pre -ordered !

    • Expensive convert your VR on something like a 3000$ HoloLens without its worst disavantadge?

    • JJ

      Just got the device and it sucks! After lots of trial and error and the issue, which ended up being that the USB-C was backwards hmmmm i thought that was the point of usbC was that they were dual sided but ok, the mini feed has overlapping flickering issues. After getting it into Unity i found out why and its really dumb. They render both cameras to 2 seperate planes and just atach those to two cameras in the scenes. These planes are on the same z plane so they flicker in and out while one goes in front of the other. Aside from that the spatial data it builds just falls apart and does not work unless your in a large, non reflective, dark room. It looks like the hololens when building but works sooo much worse.

  • dk

    really old possibly fake demo …..the 2 newer videos from 5 months ago r more interesting……also another thing to think about …..the vive has a usb on the top of the headset ….the rift doesn’t

    • benz145

      It is old, but I doubt it’s fake, it was very similar to the demo I saw myself.

      • dk

        the newer ones look less like a promotional video

      • Daniel Pramel

        I have the normal version of the Zed camera and it works like shown in the video. I am using it for something else but also played around with its AR sensor capabilities attached to the Rift.

      • jj

        I also have the new zed mini and unlike Daniel it does not work like the video. They clearly added CGI and are purposefully only doing certain actions to combat some of very noticeable issues that happen right away with the zed. For example the FOV is extremely small

    • JJ

      Yeah the vive usb cannot support the bandwidth required for the ZEd which needs a LOT of bandwidth so you better use the right usb port or it wont work. Also your right they faked a lot here, notice how slow the user moves his gun? because there is no hand or object tracking theyre using a third party tracker on a gun mount and covering it with the graphics.

      So do not assume this has leap motion hand tracking capabilities, and also do not assume hat you can attach a leap to the front while there is a zed on the front because they wont both fit.

      So yeah you’re adding another tether that doesn’t even track well.

  • Tcboy Hui Shyong

    what is the FoV, resolution and frame rate? is it the same as the bigger ZED?

    • JJ

      The real questions! Well the FOV is terrible, its like looking through a box the size of a tissue box at arms length. You will not find them mentioning this ANYWHERE which is deceptive. resolution and frame rate are changeable but its like 30 FPS at 720 and 10-15 at 1080 and 4FPS any higher….

  • johngrimoldy

    ‘Could do some very interesting things with this sort of AR. I could see experimenting with complete redecoration. The mind reels.

  • Firestorm185

    Wow, that looked fantastic! Can’t wait to try that out!

  • Lucidfeuer

    Wow late 2017 to get what VR headset should’ve had from the beginning…and I won’t even talk about those false advertised “mixed reality” headset from Microsoft.

    And am I dreaming or people reporting on it are not understanding and completely understating how huge and crucial for VR this is? Anyway not sure I’ll pre-order as this is strictly a programming research tool that’s not integrated with the few existing popular AR APIs and will quickly have limitation, but those willing to develop the underlaying computings vision and optics backbone necessary for real Virtual Headset finally have a hardware to.

    • Laurence Nairne

      Can always rely on you for a pick me up. I was starting to get quite down about the industry progress as a whole so thanks!

  • David Wilhelm

    Those are some expensive web cams.

  • Joe Black

    Haha! Finally something that makes sense.

  • Firestorm185

    Man, if it was even 250 Id consider it, but that’s more than the cost of the hmd itself now…

    • Firestorm185

      I’d love to see these integrated into the Leap Motion SDK, then it’d be fun to use both and have AR handtracking with replaceable hand models.

  • Joe Black

    I’m impressed with the environment mapping. Frankly that looks better than what current robotics manage with lidar.

    • JJ

      This is showing their large edition for big projects the small mini does not work this good at all.

  • dk
  • JJ

    So now that the zed has shipped and people have received it, its pretty clear that stereo labs has been deceitful and lied about the specs and capabilities of this device. They chose to elude from the fact that you’re looking through a small view box in the headsets which ruins the experience and even their lest latent setting is so laggy it causes half our office of vr developers to get motion sickness. Not to mention the handful of other issues it provided and the lack of forum or support by the stereo staff.