While Meta’s Quest has always relied heavily on cameras for tracking location of the headset, controllers, and the world around the user, developers haven’t had the same privileged access to the headset’s cameras. Earlier this year Meta gave developers the ability to experiment with direct access to the headset’s cameras in private projects; starting this week developers can now publicly release apps that make use of the new feature.

This week’s update of the Passthrough Camera API for Quest means that developers can now publish apps to the Horizon store that directly access the front-facing cameras of Quest 3 and 3S. This opens the door to third-party applications which can scan the world around the user to understand more about it. For instance, developers could add computer-vision capabilities to track objects or people in the scene, or to build a map of the environment for analysis and interaction.

For a long time this was impossible due to limitations Meta placed on what developers could and couldn’t do with the headset’s hardware. Despite computer-vision capabilities being widely available to developers on smartphones, Meta was hesitant to allow the same on its headsets, apparently due to privacy concerns (and surely amplified by the many privacy controversies the company has faced in the past).

SEE ALSO
'Path of Fury' Brings Nostalgic Kung Fu Fighting to Quest Today in First Episode 'Tetsuo’s Tower'

Previously, third-party apps could learn some information about the world around the user—like the shape of the room and objects within it—but this information was provided by the system in a way that prevented apps from directly seeing what the cameras could see. This made it possible for developers to build mixed reality applications that were, to some extent, aware of the space around the user. But it made some use-cases difficult or even impossible; for example, tracking a specific object held by the user.

Last year Meta announced it would finally unlock direct access to the headset’s cameras. In March, it began offering an experimental version of the capability to developers, allowing them to build apps that accessed the headset’s cameras. But they weren’t allowed to publish those apps to the public, until now.

The company has also specified the technical capabilities and performance of the cameras that the developers can access on Quest 3 and 3S:

  • Image capture latency: 40-60ms
  • GPU overhead: ~1-2% per streamed camera
  • Memory overhead: ~45MB
  • Data rate: 30Hz
  • Max resolution: 1280×960
  • Internal data format YUV420

Meta says that a developer’s use of camera data on Quest is covered under its Developer Data Use Policy, including a section on “Prohibited Uses of User Data,” which prohibits certain uses of data, including to “perform, facilitate, or provide tools for surveillance,” and “uniquely identifying a device or user, except as permitted [in the policy].”

Newsletter graphic

This article may contain affiliate links. If you click an affiliate link and buy a product we may receive a small commission which helps support the publication. More information.

Ben Lang
Ben is the world's most senior professional analyst solely dedicated to the XR industry, having founded Road to VR in 2011—a year before the Oculus Kickstarter sparked a resurgence that led to the modern XR landscape. He has authored more than 3,000 articles chronicling the evolution of the XR industry over more than a decade. With that unique perspective, Ben has been consistently recognized as one of the most influential voices in XR, giving keynotes and joining panel and podcast discussions at key industry events. He is a self-described "journalist and analyst, not evangelist."

  • That's cool!

  • Christian Schildwaechter

    Even though I have wanted camera access for years for things like QR/fiducial marker tracking, Meta's timing is right. Had they allowed access to the tracking cameras on Quest 1/2, there inevitably would have been an outcry that this allows spying inside the users' home (which it does). People still decry Amazon Alexa as spy devices (which they could be abused as), swearing to never let them into their home. So for Meta with a history of horrible data abuse to introduce a way that allows to scan the whole room while the user moves around, could have caused a PR disaster.

    But in between then and now we got usable passthrough that meant the camera data was no longer only used in a hidden way by figuring out the players position through analyzing room geometry, but openly in the users face. And we got other HMDs like the AVP that emphasize mixing virtual and real objects, obviously needing to "see" the real world. And smartglasses that are actually useful thanks to recognizing things seen by the camera. Plus developers for years publicly whining that they need camera access for any AR use that is now widely regarded as the future of XR.

    Users will be asked to allow access, but just like the technical camera access, that would have been possible on Quest 1 too. So even though nothing about the privacy issues has really changed, and the Passthrough Camera API could still be abused to silently send data about the personal environment, user profiling or blackmail ("we scanned your room and found XYZ, send bitcoins or we inform your spouse/employer/the cops/world"), this is now mostly welcomed as a good move. So technically this comes rather late, but still at the right time.

  • STL

    I never understood why I can’t record 360 degrees video or spatial video with the Quest 3 and watch them later.

    • sfmike

      Because they have that bugaboo that someone's privacy will be offended and they might have another worthless law suit.

  • Nevets

    Could this allow Gaussian splats to be recorded or would the quality be poor?