If you watched the WWDC  2024 keynote, you’ll have seen the announcement of VisionOS 2 and some highlights of what’s coming to the Vision Pro headset. But there’s lots more that Apple didn’t actually show on stage. Here’s everything we know so far.

Beyond what Apple showed of VisionOS 2 in the WWDC keynote, there’s a lot more coming when the update launches this Fall.

We’ve scoured everything Apple has released about the forthcoming Vision Pro update to get a comprehensive picture of everything that’s coming. Let’s jump right in.

Photos App Upgrades

The Photos app is getting additional enhancements in VisionOS 2. Although the iPhone can already capture spatial photos and videos, you need to switch to a special mode during capture. The Photos app will be able to take existing photos and convert them into 3D photos to view on Vision Pro. As far as we know, this will not be an option for videos.

We’ve seen similar 2D to 3D conversion tech from other companies in the past, but with mixed results. It will be interesting to see how well Apple’s approach works.

The Photos app is also getting proper SharePlay support, including Spatial Personas. While it was possible previously to stream the Photos app as a flat window with friends over FaceTime, now the app will support sharing full spatial content (photos and videos) and panoramas with other Vision Pro users.

Image courtesy Apple

Videos in the Photos app can also now be trimmed just like on iPhone and iPad.

Mac Virtual Display Upgrades

At launch, one of Vision Pro’s most useful features was Mac Virtual Display, which made it effortless to stream your Mac screen right into the headset. And while you could get a nice big monitor, it was stuck in one aspect ratio.

While many asked for support for multiple virtual monitors, in VisionOS 2 Apple opted to add new ‘wide’ and ‘ultrawide’ monitor options instead.

From a screen real estate standpoint, Apple says, “it’s the equivalent of having two 4K displays sitting side by side,” about the ultrawide mode.

New Gestures for Easier Navigation

With how often you open the app menu and Control Center while using Vision Pro, we were surprised how clunky these actions were on the headset.

Apple heard that feedback loud and clear, and has added two new gestures to make this easier in VisionOS 2.

City Builder 'Tropico' Comes to Quest, Letting You Become El Presidente of Your Own Banana Republic

To open the home screen users can turn their hand palm-up and pinch their fingers. A palm-down pinch will open the clock and make it easy to get to notifications and the Control Center where many helpful tools (like screen casting and recording) are found.

Customizable App Menu

A simple feature missing at launch; VisionOS 2 will allow users to rearrange their app menu, just like they would expect on iPhone and iPad.

You’ll also be able to pull apps out of the ‘Compatible Apps’ folder to put them in the main app menu for easier access.

Immersive Background Improvements

Image courtesy Apple

VisionOS shipped with a handful of beautiful immersive background environments, but also several tantalizing “coming soon” bubbles that we’ve been waiting patiently for.

VisionOS 2 will bring a new Bora Bora environment, taking users to an island in the South Pacific.

‘Cinema mode’ puts movies on Vision Pro onto a massive screen that naturally merges with the selected immersive background. But at launch this was restricted to apps like Apple TV.

With VisionOS 2, web videos in Safari will have the option of being presented in the same way. Apple says this will work for YouTube, Netflix, and Amazon videos out of the gate, and presumably web developers will be able to enable this functionality in other players too.

Users will also now be able to tilt videos upwards, while in cinema mode, to make reclined viewing more comfortable.

Keyboard Visible Through Immersive Backgrounds

Image courtesy Apple

Speaking of Immersive Backgrounds, at launch Vision Pro had excellent hand occlusions which brings your own hands into immersive environments. Unfortunately if you were using a Magic Keyboard, they keyboard itself would disappear when in an immersive environment, even though your hands would show on top of where it should be.

VisionOS 2 makes the natural improvement of showing the full keyboard even while an immersive background is visible.

Support for Third-party Mice

At launch Vision Pro could only be paired with Apple’s Magic Trackpad. But regular mice weren’t supported (not even the company’s own Magic Mouse).

VisionOS 2 will add support for the Magic Mouse and third-party mice too. Presumably this will cover third-party mice and trackpads.

Guest User Enhancements

It’s always fun to show people Vision Pro for the first time. And the built-in Guest User feature makes this easy. Unfortunately it takes a minute or two for the Guest User to calibrate every time they don the headset.

In VisionOS 2, Guest User can remember the last guest’s calibration data for up to 30 days. So if they need to take a break or you want to grab the headset for a minute to navigate them to a new experience, they can put it back on and skip the calibration step.

Watch Multiple Sports Games in Apple TV

In VisionOS 2, Apple TV will allow users to watch up to five different games at the same time—between MLS and MLB.

Persona Enhancements

Apples Persona avatars have been steadily getting better since the launch of Vision Pro, and VisionOS 2 promises more improvements still.

Apple says users can expect “more accurate skin tones and vibrant clothing colors.” It also promises more natural hand movements.

'Everybody's Golf' Studio Tees Up the Competition on Quest in 'ULTIMATE SWING GOLF'

And if you’re using your Persona as a ‘webcam’ (ie: a windowed view, not a Spatial Persona) you’ll no longer be stuck using the default background that every Vision Pro user is stuck with. With VisionOS 2 Apple says you’ll be able to choose “a variety of backgrounds,” hopefully including choosing your own photos.

Look-to-Dictate in Messages

Look-to-Dictate is a convenient part of Safari, where you just look at the microphone icon in the URL bar, and it prompts you for voice input (skipping the keyboard).

VisionOS 2 will bring the same feature to Messages to make it easier to send texts hands free.

AirPlay to Vision Pro

It’s always been possible to cast content from Vision Pro, but not to cast content to Vision Pro from other AirPlay devices.

VisionOS 2 will make this small but obvious improvement, meaning you’ll be able to cast from your iPhone, Mac, and other AirPlay compatible devices, and see the video pop up in your headset.

Mindfulness App Breathing Detection

Image courtesy Apple

Mindfulness is Apple’s first-party meditation app for Vision Pro. In VisionOS 2 the app will now be able to detect your breathing rate and use it to match the app’s visualizations. This is presumably done using the headset’s microphones.

Train Support for Travel Mode

Image courtesy Apple

Travel Mode in VisionOS 2 will explicitly support use in trains. The mode changes the headset’s tracking behavior to ensure it tracking works accurately even while being in a moving vehicle.

Panorama Viewing on the Web

In the Photos app, panoramic photos can be stretched around you to experience them in an immersive way. That same feature will now work with in Safari for panoramic photos hosted on the web. We don’t know at this point if the headset will automatically detect panoramic photos or if webpages will need to declare them as such in order to use this feature.

Apple Music Karaoke

The Apple Music app in VisionOS 2 will get support for collaborative playlists between friends, and Apple Music’s ‘Sing’ feature which works like karaoke. It sounds as if friends will even be able to use Sing together through FaceTime.

Live Captions

Image courtesy Apple

The Live Captions feature is also coming to VisionOS 2. This acts like ‘real time subtitles’ by showing in the headset a written version of the words being spoken around you, or being played through the headset in audio or videos. Primarily intended as an accessibility feature for those with hearing disabilities, the feature may one day support real-time translation too.

Improved Quick Look

Quick Look is the easiest way to show realistic 3D models in augmented reality at scale—even directly from a web page. VisionOS 2 will bring improvements to Quick Look, like being able to anchor objects to surfaces and change colors and materials without loading additional models. Apple envisions this as being especially useful for previewing how a piece of furniture might fit in your room.

Apps Will Remember Their Position Between Reboots

It’s impressive that you can ‘leave’ and app sitting on one side of your house, walk away, then come back and find it still hanging out right there. But unfortunately if you power down your headset the app will forget its positions.

In VisionOS 2, apps can remember the position even between reboots. However, our understanding is that recenting the headset will still recall all apps to appear in front of the user, ruining any large-scale app arrangements you may have set up.

Communication Safety in VisionOS Apps

VisionOS 2 will bring Apple’s Communication Safety feature to the headset. The feature, intended primarily for children,” gives you the option to blur sensitive photos and videos you receive before you choose whether to view them.”

There’s a bunch of other changes and improvements coming for developers too, stay tuned for more coverage on that front.

Newsletter graphic

This article may contain affiliate links. If you click an affiliate link and buy a product we may receive a small commission which helps support the publication. More information.

Ben is the world's most senior professional analyst solely dedicated to the XR industry, having founded Road to VR in 2011—a year before the Oculus Kickstarter sparked a resurgence that led to the modern XR landscape. He has authored more than 3,000 articles chronicling the evolution of the XR industry over more than a decade. With that unique perspective, Ben has been consistently recognized as one of the most influential voices in XR, giving keynotes and joining panel and podcast discussions at key industry events. He is a self-described "journalist and analyst, not evangelist."
  • They showed almost all of these at the keynote …. lol
    []^ )

  • Wow, this is a lot!

  • Christian Schildwaechter

    Knowing how many people struggle with their hearing aids, with higher end models being significantly more expensive than AVP, the live captions feature alone might sell a lot of (future, lighter) HMDs. Simply because it could get people "back into the conversation", something that current hearing aids that try to filter/enhance the sound to whatever hearing the wearer has left often fail to do.

    Having the transcribed text as a backup while still being able to look directly at the person you are talking too will help a lot. With six microphones and using spatial audio models, AVP could also triangulate the source of any sound in the room and e.g. only amplify speech coming from the direction you are looking at, and lower all other noise.

    If they manage to make like captions also usable with all media available on AVP, esp. those that don't offer any type of captions themselves, this could seriously improve the life of many people. It's kind of overkill to use a very powerful XR HMD mostly as a hearing aid, but simpler/lighter glasses like displays even with miserable FoV currently simply don't offer these features. AVP even at its current price would be competitive due to the insane prices of hearing aids. The main issue would be that it is way too heavy and impractical to be used as a (reliable) full time replacement or outside.