With Apple’s increasing interest in VR on Mac, the company is adding essential features to its new Metal 2 graphics API to support high performance VR on MacOS.
Because VR demands high-powered graphics and extremely low latency, specially tuning the rendering pipeline between the GPU and the headset is crucial to maintaining a comfortable and performant VR experience. Many of the key tweaks to the VR rendering pipeline have been established on other platforms, and Apple is bringing many of them to the updated Metal 2 API which the company says now “provides powerful and specialized support for Virtual Reality (VR) rendering.”
At the ‘VR with Metal 2’ session during Apple’s WWDC 2017 conference last month, Apple GPU Software Team member Rav Dhiraj detailed the new features in Metal 2 for VR.
Single-pass Stereo
Since VR headsets show a 3D image, they need to render an individual view for each eye; you might think this would require double the rendering work, but with some smart rendering techniques you can be much more efficient. Single-pass stereo allows the GPU to render to the left and right eye with a single draw call instead of one for each eye.
Metal 2 now makes this possible with what Apple calls the Viewport Array feature which allows “per-primitive viewport selection in the vertex shader.” Instead of rendering the left eye image and then the right eye image, developers define a texture that’s the resolution of both eyes next to each other, then render across the entire texture in one pass. Telling the renderer which eye each primitive belongs to ensures that it gets rendered in the correct eye-region of the texture and with the correct offset to account for the slightly different perspective of each (due to the distance between the eyes).
Direct-to-display
Direct-to-display (also known as Direct-mode) allows the GPU to more directly access the VR headset by bypassing parts of the pipeline that would be necessary only for traditional monitors.
Without Direct-to-display, the operating system would see a VR headset like any other monitor and simply mirror or extend the operating system view onto it, including modifying the rendered image with any OS-induced post-processing. With Direct-to-display, the operating system doesn’t identify the headset as a monitor, preserving the existing window layout and display arrangement. This allows a VR compositor (which warps the initially rendered image and prepares it for display on headset) to directly present the rendered image to the headset’s display without the OS introducing any additional latency, performance hits, or image tampering.
Not only does Direct-to-display reduce the complexity of the VR rendering pipeline, it also makes the end-user experience better because developers and users don’t need to fuss with ‘where’ on the OS’s desktop the VR application is displayed—instead it automatically ensures the rendered image lines up perfectly with the VR headset’s display and without any interference from the OS.
– – — – –
Metal 2 will debut with MacOS version ‘High Sierra’ which is presently in beta for developers and due to launch later this year, supporting all Macs compatible with the present ‘Sierra’ version.
Developers building VR apps with tools like Unreal Engine or Unity won’t need to worry about the above features as the MacOS VR integrations of those engines will take advantage of them by default. Developers wanting to building native VR applications or integrate with existing proprietary game/rendering engines on MacOS will want to specifically adapt support for these new VR specific features.
Both features and more are covered in detail during the WWDC session, including an overview of best-practices for VR rendering that will be useful to developers building VR apps from scratch.