With Quest 3 now officially announced, Meta is emphasizing the device’s improved MR capabilities.

Meta CEO Mark Zuckerberg took to Instagram to share a first look at mixed reality gameplay on Quest 3 which was announced yesterday.

 

View this post on Instagram

 

A post shared by Mark Zuckerberg (@zuck)

The video shows the headset’s full color passthrough MR mode, which allows it to present a view of the outside world while selectively adding virtual content to the scene.

We also see some shots of virtual objects attached to the wall, like a glass window into an undersea world, or a zombie jumping through a window into the room to attack the player. While Quest 2 and Quest Pro have done the same in the past, Quest 3’s new depth sensor should make attaching virtual objects to walls, floors, and ceilings more convincing thanks to a more precise map of the world around the headset.

We also see Meta CTO Andrew “Boz” Bosworth jump into the action, showcasing a co-presence experience where both Zuckerberg and Bosworth battle each other virtually but in the same physical space.

Beyond Quest Pro

It’s difficult to tell from the footage how Quest 3’s passthrough resolution compares to Quest Pro. However, it’s notable that the footage doesn’t show any of the obvious color fringing that was an artifact of Quest Pro’s passthrough architecture, which used multiple black-and-white cameras that were fused with the color from a single RGB camera. That ought to be solved now that Quest 3 will include two RGB cameras which will allow stereoscopic capture of color information, rather than monoscopic like with Quest Pro.

Another common artifact of Quest Pro (and Quest 2) passthrough is the warping of objects (especially hands) that are close to the headset. This is caused by a breakdown of the computer-vision depth estimation which struggles with near-field objects, especially when they’re moving.

It’s difficult to tell from the footage we have so far, but there’s a good chance that Quest 3 significantly reduces these passthrough warping artifacts thanks to its included depth sensor. Whereas Quest 2 and Quest Pro estimate the distance to objects and surfaces around the headset with computer vision, Quest 3’s depth sensor will provide much more reliable distance measurements which the system can use to judge how far it should render each part of the scene.

SEE ALSO
This is the Best Look Yet at What It's Like to Use Vision Pro

It will be interesting to see if the prior issue with color fringing on Quest Pro manifests in the same way with depth. With a single depth sensor, the headset only has a monoscopic depth view, whereas it will have a stereoscopic visual of the real world. Ostensibly the stereoscopic view of the world will be projected onto the depth map, and ‘depth fringing’ may occur around near field objects for the same reason that we saw color fringing on Quest Pro.

Newsletter graphic

This article may contain affiliate links. If you click an affiliate link and buy a product we may receive a small commission which helps support the publication. More information.


Ben Lang
Ben is the world's most senior professional analyst solely dedicated to the XR industry, having founded Road to VR in 2011—a year before the Oculus Kickstarter sparked a resurgence that led to the modern XR landscape. He has authored more than 3,000 articles chronicling the evolution of the XR industry over more than a decade. With that unique perspective, Ben has been consistently recognized as one of the most influential voices in XR, giving keynotes and joining panel and podcast discussions at key industry events. He is a self-described "journalist and analyst, not evangelist."
  • Dragon Marble

    A much underappreciated fact is that the PSVR2’s pass-through has zero warping and is perfectly depth correct. It does not have a depth sensor, and the cameras are placed in odd positions. It’s quite impressive Sony was able to accomplish this. Maybe all you need is that computing power from PS5.

    I assume PSVR2 can do mixed reality too, even though that’s not going to be the focus.

  • Jan Ciger

    The warping problem on Quest 2/Pro most likely isn’t because of the distance estimation/tracking failing up close (controllers are tracked at those distances fine) but because the image the user sees is synthetic – it is an artificial image calculated out of variously warped and combined images of the tracking cameras + RGB camera, not something any of the cameras actually see.

    When the object gets too close, one or more cameras can’t see it anymore, causing computational issues – part of the image data is missing and has to be “interpolated”, for lack of a better word.

    Furthermore, the extreme fish-eye distortion of the tracking cameras is more pronounced and more difficult to compensate for when the objects are close than when they are far, again, causing computation problems.

    It is pretty much the limitation of the mathematical model of the lenses and cameras showing when they are being pushed to do things they were not designed for – displaying objects very close to them.

  • MarcDwonn

    LOL, who cares about passthrough… Higher FOV and better clarity is what people want these days from a headset. And a good PCVR connectivity with low compression artifacts, and even DisplayPort over USB-C if technically possible.

    • eadVrim

      I care about passthrough

    • Jonathan Winters III

      Well you’ll be disappointed (as I am) to know that Quest 3 has the same FOV as Quest 2. Nevertheless, love both.

    • Rupert Jung

      … and better black levels, please. No more LCD. :)

  • Christian Schildwaechter

    TL;DR: if there is a killer app for Mixed Reality, it will be not having to clean up your play space every single time you want to use it.

    There is a single scene in this video that I would consider to be a demonstration that MR on Quest 3 can provide an actual improvement that isn’t just a gimmick: Andrew Bosworth opening the door.

    They show some of the interaction with the furniture that are already possible on Quest 2/Pro by mostly manually setting up the environment. Which then allows some zombie to jump out of the wall, onto the sofa, and from there onto the floor. Which is a gimmick, and will still still be a gimmick if the Quest 3 fully automatizes the setup thanks to the integrated depth sensor.

    What is not a gimmick is proper live detection of changes in the environment and being able to always show obstacles without ripping you completely out of VR. Zuckerberg and Bosworth move around in a rather large room with a lot of space, always staying within the area cleared of obstacles, but many VR users simply don’t have a large space permanently free of objects, so they either have to deal with a very small guardian or move the furniture every time they want to enter VR, if there is even enough space to move the furniture to. Which adds a whole lot of friction.

    Real time environmental scanning with the Quest 3 depth sensor could help here. I’d still not recommend starting a boxing match with a lot of back and forward movement in a room full of clutter, as at some point you’d inevitable trip and injure yourself. But there are a lot of slower, explorative room scale titles like escape room puzzles that would really gain from any extra space and that could blend in real world obstacles. This would of course be somewhat immersion breaking, but still a lot better than triggering the guardian all the time because you had to define a very tiny play area free of objects like the sofa, even though the air space above it would be fully usable. And in a case of doubt you can push the chair or table around in your play space whenever it is in your way, but at least you don’t have leave VR for that or set up the guardian/environment again, as the new obstacle position would be immediately detected and integrated.

    Passthrough is very useful for setup and small real world interactions like finding a cup. Most of the other demonstrated use cases are way overblown and not something you’d really need or use a lot, like a virtual aquarium with a white shark. Sure, you can play Cubism and watch out for your cat or CTO entering the play area without you noticing, and this is probably an essential feature for parents of small kids. For most others this is a problem that can be solved by closing the door and other household members using common sense.

    The lack of free space problem is much harder to solve, and I’d expect that a significant cause for the low retention of VR is the hassle required to first guarantee a safe play area whenever you want to use your headset. Console gaming works fine within a domestic disaster area, at most you have to push some stuff off the sofa and remove anything within direct line of sight to the TV. But without a lot of spare room, VR requires you to become the child your parents probably always wished for, the child that cleans up its room every time before it is allowed to play.

    • Dragon Marble

      I think you are missing the point. You would start the boxing match with pass-through on so you can use footwork with confidence. That is applicable to any physical game, some with straightforward adaptation (Blaston, Eleven, e.g.), others with some rework and re-imagination (Crisis Brigade, Gorn, e.g.).

      And the real game changer is one you add another person. I can’t wait to get a Q3 to play co-presence games with my kids (assuming it works with my Pro). Mixed reality removes one of the biggest friction point of VR: isolation, turning the games into family activities.

      P.S. I do want a shark tank (and other cool decorations) in my living room, and change lighting depending on my mood.

      • Christian Schildwaechter

        My whole argument is about being able to play in a space with physical obstacles, so my boxing example isn’t about a lack of confidence or balance, but about accidentally tripping over the coffee table while moving backwards, trying to avoid a punch. Which would happen with or without passthrough. What I am suggesting pretty much excludes all games that require you to physically move quickly around the room while focusing on an opponent, as this would most certainly lead to injuries. I see using MR with the Quest 3 depth sensor as an option to somewhat safely play (some slower) room scale games in a place where there isn’t enough physical room, thus opening VR to a lot of people that simply lack the room for a properly sized play area.

        I’d agree that the isolated experience is what keeps the majority of people away from VR, but honestly I don’t see MR really changing that. You get the passthrough option, allowing you to see the environment, but that doesn’t really make it much more social beyond you being able to turn towards another person you are talking to in the same room. There were a couple of games for PSVR allowing for one person to play in VR together with others playing on the TV, but the Quest 3 doesn’t really allow for that, the VR experience will still be pretty much solitary.

        The Quest Pro will probably allow for more/better interaction with others by simply removing the facial interface and letting you see your actual environment in the periphery, which neither requires games to be designed so that the background can be replaced with passthrough, nor does it come with the lag, distortion or lower resolution that is inevitable with any passthrough solution when compared to your eyes. Passthrough helps to watch your cats/kids, but other than watching the stream of your VR session they will still be limited to watch you fumbling around in the air in a world they cannot see. I doubt that this is what those that didn’t like the isolated VR experience so far are looking for, but I’m guessing here, we’ll have to wait and see if people will use it and what types of games will work with MR in the first place. So far it is mostly puzzles with floating pieces, not exactly why most people get into VR.

        And sure, you can have your shark tank, but I doubt that you will sit in you living room wearing a Quest 3 for hours watching it. It will make for a nice background when you use an app that works with passthrough, but not a reason to wear the headset itself after an initial tryout phase. Which is why I consider it a gimmick.

        • Dragon Marble

          Oh, when I say “use footwork with confidence” I mean you are not afraid to bump into things.

          Co-presence is much more than seeing your surroundings. It allows two or more to be in the same game while also in the same physical space, so you’ll be interacting with the actual person rather than the avatar. It’ll require multiple headsets in the household, so there’s a cost hurdle there. But that’s something I have been wanting to do since they introduced this feature in Quest Pro — except that I can’t justify buying two Pros for exactly one app that supports this.

          • Christian Schildwaechter

            I probably didn’t get your meaning of “footwork with confidence” because I never fully achieved this myself. I traveled to gamescom 2015 for one of the HTC Vive prerelease demo spots, and it took me roughly a minute to step on the cable with one foot and then against the now tightened cable on the floor with the other foot, ripping out the connector box and requiring everything to be restarted. Also hit the carpet covered walls a couple of times. I got one of the last demo spots, and apparently during the whole week nobody else messed it up this way (I asked). So maybe I have an inert talent for causing accidents in VR and therefore an increased interest/need for ways to be made aware of obstacles.

            Your description of co-presence with two people wearing HMDs in the same physical space puts you firmly at the other end of the VR user spectrum compared to those I was referring to. Not only do you have enough space for two people to interact in, you also already have a rather expensive headset and are willing to add another one for the gained experience in a few select apps, as so far co-location isn’t even supported. This places you firmly in the enthusiast corner with the necessary drive and means to fully utilize it. My perspective was more on casual users that may be new to VR, use it only occasionally and have to integrate it with a sub-optimal setting. Those struggling to make it work for just one person.

            This may be partially a cultural problem. I live in the center of Berlin, where many people inhabit apartments the size of a typical US suburban living room. When demoing the Quest 2 at friends, it was often immediately clear that they would never go for it, simply because it would require regularly rearranging half their home. Having a mode that allowed for a more efficient use of the space by e.g. blending in translucent boxes into regular VR games like RE4 instead of relying on virtual walls from the floor to the ceiling to mark the free space could make them playable without having to stick to stationary mode and having to teleport even short distances, as a lot of people cannot handle smooth stick based motion. So enabling rather basic VR functions for existing games, very far from more advanced ideas like multi-person co-presence or future non-casual games that manage to integrate MR passthrough in a way that doesn’t conflict with taking place in another world.

    • Hussain X

      If I lived in a domestic disaster area, I’d still be using VR/MR for gaming over console gaming on a monitor/tv. I can play sitting down VR games with motion controls and gamepad, VR modded flat games such as Luke Ross’ and Praydog’s mods, even using reshade to play flat games in 3D. Worst comes to worse, play flat games in VR in 2D (maybe not competitive games) but on a giant curved screen the physical equivalent of which the domestic disaster zone won’t have space for nor the wallet have the money to buy one. All above scenarios will give a much immersive experience than console gaming can on a smaller tv/monitor. The potentially low retention of VR for those living in domestic disasters zones is probably more to do with a lack of sit down VR games, and friction and lack of know-how of VR modding, setting up reshade etc, and needing a beefy pc. Plus you can quickly move that giant tv to your living room, to your bedroom, on to your bed, and put it away out of existence with one click when you’re not using it, freeing up space in that domestic disaster zone. I’m sure if you could play ps5 flat games on a giant virtual screen and in 3d on the psvr2 without any added friction performance issues, PSVR2 owners won’t have issues with VR retention. Flat gaming in VR/MR will offer a far superior immersive 3D gaming experience than on a tiny TV/monitor for the guy living in a domestic disaster zone.

      • ViRGiN

        stop masturbating about praydog and luke ross you pcvr elitists. lmao!

      • Christian Schildwaechter

        The potentially low retention of VR for those living in domestic disasters zones is probably more to do with a lack of sit down VR games, and friction and lack of know-how of VR modding, setting up reshade etc, and needing a beefy pc.

        Pretty sure that every single point you mentioned in fact adds a lot of friction, and if VR becoming more popular requires regular users to set up mods and ReShade themselves, it will most certainly never go mainstream. That’s the very core of the problem: you can already have an amazing experience in VR with crispy sharp graphics, epic open worlds, full body tracking etc., but you basically have to become an expert first. The reason the Quest 2 was so successful wasn’t only the low price, but also the convenience of instantly working without any complex setup required from casual users.

        And playing on a large virtual screen is an argument if what you are looking for is a large screen. I own a Steam Deck, basically the opposite of that, and I prefer it for a lot of titles because it is pretty much the pinnacle of convenience: works anywhere, doesn’t need to be plugged in, can easily be carried everywhere. And thanks to being based on Linux/Steam OS with a system wide instant suspend/resume function, it takes a literal second to stop playing and another second to later continue at the same spot. Despite the low performance and tiny screen all this has caused a massive shift in my gaming, with the PC being relegated to those games that really benefit from a large 4K screen, and even these I now often stream to the Steam Deck.

        Convenience is what caused mobile games to quickly dominate the whole gaming market, now generating 50% of all the revenue and still rising. VR enthusiasts treasure the immersion that VR brings and that will never be matched by any flatscreen game, but immersion isn’t as important to the majority as convenience, with friction being the antipode to convenience. If you treasure the immersion or having a large screen you can use where ever you want, wearing a not exactly comfortable headset and even occasionally making space for the games that simply need you to move around will be an acceptable price. But so far those willing to deal with it are a very tiny minority, and for VR to grow and get enough users to justify developing more titles and integrating seated play options it has to get less intrusive first. Which means less friction, not everybody figuring out how to configure ReShade or what ReShade is in the first place.

        • Hussain X

          “Pretty sure that every single point you mentioned in fact adds a lot of friction”.

          This is what I’m saying. Too much friction for those wanting to do sit down VR. Most VR releases are standing up, needing space type games. Some people don’t have that space readily available, or maybe tired after a hard day’s work to want to stand up to play, and maybe would prefer to do sit down VR gaming. But there is either a lack of ground up sit down VR games, and the alternative of reshade, mods, etc, have a lot of friction and need a powerful gaming pc.

          The reason for my original reply was because you said:

          “The lack of free space problem is much harder to solve, and I’d expect that a significant cause for the low retention of VR is the hassle required to first guarantee a safe play area whenever you want to use your headset.”

          All I’m saying is low retention of VR may not be about space. If players had either high quality ground up sit down VR games, or could play traditional games with VR mods, reshade, or even just on a giant 2d virtual screen without the friction it currently has, VR would get used a lot more even if the player had no space or can’t be bothered to create a safe play area.

          Imagine like your steam deck or better comparison, Project Q from Sony PlayStation. You put your Quest 2 headset on, one click connect to pc, one click open a AAA game on a giant virtual screen in 3D – ie 2 clicks and less than 10 seconds you’re playing a AAA game in 3D on a giant virtual screen. If a gamer could have that frictionless choice and convenience, he’d probably play his games on a VR virtual 3D screen than on a small 2d TV. So what I’m saying is, low VR retention may not be an issue with space. Only that for now, it needs the space due to the types of VR content coming out (frictionless but needing space), and because of a lack of frictionless, convenient alternatives of using VR sat down (frictionless and not needing space).

          • Christian Schildwaechter

            Okay, I get where you are coming from. I don’t think I agree that providing a seated VR option would really solve the issue, as this may be a good solution for playing non-VR titles and a welcome comfort option for those that currently don’t want to play standing up, but early on VR users made it pretty clear that being able to move around the room and physically explore the surrounding adds a lot to the experience. And many seated-only VR games that weren’t simulation of seated activities or those only supporting game pads were punished with bad ratings and harsh reviews.

            So I’d say the space requirement is still a fundamental issue and will always add a lot of friction. As mentioned I don’t consider even stationary mode and teleporting as a good alternative for room scale, and this would still be more immersive than sitting down. Seated VR is a lot better than nothing and may even be the preferred way to play for lots of people, but many are looking more for the opposite, hoping that we get omnidirectional treadmills to get rid of all movement limiting play space areas.

            I agree with the rest of your argument about more people using VR more if the friction could be reduced, for example by making playing regular games on a large screen in the Quest 2 a one-click solution. I have to refer to the Steam Deck again as a reference, where Valve provides special configurations for games to make them run better with their Proton compatibility layer, lets users create and share special input configurations for different games and even allows to make game-specific hardware configurations like disabling cores, limiting TPD etc. This way a user doesn’t have to know anything about the configuration and instead can play games in an optimized configuration right from the start, or pick one that suits them better.

            Something similar would no doubt be possible with VR games and mods. So you’d basically say you want to play a game seated with with tracked controllers with your current hardware configuration and streaming to a Quest, and it would automatically pick the best configuration. The main problem is that I don’t see who would sponsor this effort. Valve does it for the Steam Deck to get people to stick to the Steam platform and reduce the amount of service queries and complaints they’d otherwise get, and it works because they have control over the game distribution, hardware, SteamOS and the configuration, meaning they can pretty much ensure that the configurations will work.

            They don’t necessarily have a lot of incentive to do the same for VR mods which are pretty much all developed outside of Steam and running on non-Valve hardware. And Meta mostly tolerates PCVR, I doubt that they would be willing to help with anything that doesn’t drive software sales on the Quest store. Virtual Desktop might be another potential source for pooling VR optimized configurations, similar to Ralf Ostertag providing game configurations for vorpX. Some of the mods are locked behind Patreon memberships, and even on open sites like Nexus mods you need a payed subscription to automatize the installation and configuration of mods, making it rather difficult to come up with a solution that could unify configurations without adding another layer of complexity from having to deal with multiple separate platforms.

            So while making the configuration of all sorts of VR usage, from seated VR to streaming 2D games would technically be possible and would no doubt reduce friction a lot, thus making VR much more accessible, for the time being this seems to be a mostly theoretical option. Going back to the start of the argument, with the depth sensor Meta could provide a system wide way to allow VR to be workable in a lot smaller spaces with obstacles. And it wouldn’t require any extra activity from developers, many of which won’t go beyond making sure that their games work with a stationary guardian too, or involve anything not under Meta’s control or in their interest. It would in no way be as flexible as what you have in mind, but a lot easier to achieve.

    • eadVrim

      Playing virtual games like tennis table, mini golf or many other board games with another person in the same room/house is a great added value can be provided by the passthrough.
      And we could see more than one Quest in same home in the futur.

      • Shy Guy

        It will be even better if you can ‘merge’ your table with that of someone else who is in their own home far away sitting at their own table, and you can play together as if you were both in the same room.

    • kool

      This is an American company and most Americans live in big empty houses. Most men I know at least have a man cave especially if they game. I’ve been to Europe and understand what you mean tho.

  • KungVr

    I really think this would be a good thing for VR porn. Makes it much more immersive in your own bedroom

    • Shy Guy

      Isn’t current VR porn just stereoscopic 360 degree video? There’s no real way to mask out the background, and align things with your room. Even if there was a way around that, and ignoring other furniture, your bed would need to be the same size and height as the one they filmed with for things to line up otherwise you’d end up with floating people, and feet through the floor or in mid air.

  • Andrew Jakobs

    Shame the video doesn’t load, at least not on mobile, also not on instagram itself.

    • Only trolls hide comments

      Works for me. Problem must be on your end.

  • Jonathan Winters III

    The color fringing of Quest Pro will not be present on Quest 3. That was simply an artifact of the Pro’s lack of depth sensor etc.

  • KungVr

    You can mask out the background with a green screen, that works already. When filmed it lacks the 6dof, but animated porn can look really realistic and that can track you background so the girl stay on your bed while you move around it. It kinda already works with VirtAMate and ALXR in an early stage

  • Twa Corbies

    More minigames. Meh

  • Rupert Jung

    I was really, really hoping for eye tracking as the primary advance when compared with Quest 2. Should really be standard these days for descent headsets.

    • Guest

      On some level I agree, but on the other hand, eye tracking also enables the most egregious form of privacy invading data collection. Nothing that comes before is even remotely comparable in terms of value. Apple might be decent about privacy (in the context of major corporations), but Zuckerberg?