Low-latency Rendering

Overview

The elapsed time from sensing to rendering—sometimes called ‘motion to photon’ latency—has been the subject of intense scrutiny towards creating more comfortable and immersive experiences. Latency comes from multiple sources including: how often do the sensors generate data? How quickly does that data get to the application? Are data points extrapolated into the future to create ‘predictive’ tracking? How quickly can the application render the image through the graphics stack? Can the application make ‘time warping’ corrections at the last instant?

Current Status

OSVR systematically addresses these points as follows:

  • Data rate from sensors: The OSVR HDK provides 100 Hz positional data and 400 Hz “sensor-fused” orientation data.
  • Speed in which data reaches the application: OSVR integrates with ETW (Event Tracing for Windows) which is a powerful tool for latency and performance analysis (See http://osvr.github.io/presentations/20150901-Intro-ETW-OSVR/for a brief tutorial). ETW helps optimize the full software stack—from the game to the sensors—towards achieving optimal performance.
  • Predictive tracking: OSVR currently includes predictive orientation tracking. When using the OSVR HDK, this is derived from angular velocity reports that are provided at 400Hz to the controlling PC. When using HMDs that do not provide angular velocity, the velocity can be extracted from consecutive yaw/pitch/roll reports. Predictive tracking looks 16 milliseconds into the future and reduces the apparent latency.
  • Direct render: The Sensics/OSVR Render Manager (supported on Windows/NVIDIA platforms and includes a Unity Plugin) provides optimal low-latency rendering on any OSVR-supported device. It includes:
    • Direct to HMD Mode: Enable an application to treat VR Headsets as head mounted displays that are accessible only to VR applications, bypassing rendering delays typical for Windows displays. Direct to HMD Mode is available for both Direct3D and OpenGL applications.
    • Front-Buffer Rendering: Renders directly to the front buffer to reduce latency.
  • Time warping: OSVR includes Asynchronous Time Warp as part of the Render Manager. It reduces latency by making just-in-time adjustments to the rendered image based on the latest head orientation after scene rendering but before sending the pixels to the display. It includes texture overfill on all borders for both eyes and supports all translations and rotations, given an approximate depth to apply to objects in the image.
SEE ALSO
Vision Pro is Hands-down the Best Movie Experience You Can Have on a Plane

See Also: New OSVR HDK v1.3 Headset Gets 120Hz OLED Display, New Optics – Ships in October

Future Plans

The OSVR team is working to expand the range of supported “Render Manager” graphics cards to include AMD and Intel. We are also looking to add Render Manager capabilities on Android and other non-Windows platforms by collaborating with graphics vendors for these platforms.

With regards to the Render Manager itself, in the very near future, the following enhancements will be released:

  • Integrated Distortion Correction: handling the per-color distortion found in some HMDs requires post-rendering distortion. Today, OSVR performs this on the client/application side, but moving distortion correction into the Render Manager provides additional benefits. The same buffer-overfill rendering used in Asynchronous Time Warp will provide additional image regions for rendering.
  • High-Priority Rendering: increasing the priority of the rendering thread associated with the final pixel scan-out ensures that every frame is displayed on time.
  • Time Tracking: indicating to the application what time the future frame will be displayed lets it render the appropriate scene. This also enables the Render Manager to do predictive tracking when producing the rendering transformations and asynchronous time warp. The system also reports the time taken by previous rendering cycles, informing application when to simplify the scene to maintain an optimal update rate.
  • Add Render Manager support in additional engines.
  • Use ETW to perform additional engine and plugin optimizations.
  • Design of a plugin-based API for advanced rendering capabilities. This would allow open-source release and public development of a cross-platform Render Manager stack with the ability to load vendor-specific code (which may be closed-source if required by those vendors).

Operating Systems

Overview

VR/AR products come in many shapes and forms and reside on several different computing platforms. The obvious examples are PC-based VR, which typically uses Windows, and mobile VR, which typically uses Android phones. The goal of OSVR is to both support a wide range of operating systems as well as provide a consistent programming interface regardless of the particular operating system. In an ideal case, OS-specific details (such as graphics drivers, file system details) as they relate to creating the VR/AR experience are all abstracted by OSVR.

SEE ALSO
20 Great VR Games for Relaxation & Meditation

Current Status

OSVR currently supports the following operating systems:

  • Windows, including Windows-specific support for direct rendering.
  • Android, including the ability to use internal sensors, camera and other on-board peripherals. Currently, this uses the CrystaX NDK to build native-code applications. Unity/Android also works on top of OSVR/Android.
  • Linux: The OSVR engine has complete support for Linux and the code is tested on Linux before Windows binaries are released. Unity/Linux should be possible but has not been tested yet.

Future Plans

  • Include Android and possibly Linux binaries in Unity plugin releases.
  • Add OSX support.
  • Add iOS support.
  • Add RenderManager support to other platforms, working with platform vendors and manufacturers of graphics chips as required.
  • Validate correct operation of additional game engines for non-Windows operating systems.
  • Add plugins for OS-specific/platform-specific peripherals and capabilities.

Utilities

Overview

OSVR utilities are standalone programs that are perform useful functions in support of OSVR development and deployment. True to the OSVR philosophy, OSVR utilities are also open-sourced.

Current Status

The following utilities currently exist:

  • Distortionizer: The distortionizer is an interactive tool that helps determine the RGB optical distortion parameters of a display. Sometimes, these parameters are provided by the optical design team. Other times, they need to be measured. The output of the distortionizer is a set of distortion correction coefficients that automatically feeds into the Render Manager.
  • Latency test: This combines open-source hardware (based on low-cost Arduino components) and open-source software to provide end-to-end latency measurements.
  • Tracker Viewer: a graphical utility to dynamically display position and orientation of several tracked objects.

Future Plans

I am aware of several additional utilities under development:

  • Windows installer for the runtime components of OSVR.
  • Interactive configuration utility to allow configuring eye separation, height and other user-specific parameters.
SEE ALSO
What Any VR Game Can Learn From the 'Electronauts' Interface – Inside XR Design

High-level Processing

Overview

High-level processing modules (Analysis Plugins in “OSVR speak”) are software modules that convert data into higher-level information. For instance, a gesture engine plugin can convert a stream of XYZ coordinates to a recognized gesture; an eye tracker plugin can take live feed from a camera pointed at the eye and provide real-time gaze direction; a sensor-fusion plugin can mesh data various tracker interfaces into more accurate reports.

Current Status

The OSVR team and community have primarily been focused on building the lower-level infrastructure before adding optional processing layers on top of them. At present, two analysis plugins exist:

  • 1-Euro filter: This is a data smoothing filter that, as currently applied, improves stability of the output from a Razer Hydra.
  • Predictive tracker: Estimates head orientation into the near future as a method to reduce perceived latency.

Future Plans

A unified API for easily developing analysis plugins and allowing their configuration is in progress. Several analysis plugins are under consideration and in various stages of design:

  • Sensor fusion: to combine the output from multiple sensors into more accurate information.
  • Augmented reality: to allow detecting objects in the scene.
  • Eye tracking: to convert pupil camera images into pupil coordinates.

We are working to simplify the process of writing analysis plugins, to provide open-source examples and are very open to community contributions.

Summary

OSVR, like VR in general, is a work in progress. I am very pleased with the quality of partners, the breadth of functionality, and the level of community involvement that we have seen in the eight months since launching OSVR. Having said that, there is still plenty of work to be done: whether in supporting new devices, supporting a wider range of engines and development environments, in making devices more useful by creating analysis plugins, or in providing high-performance rendering across multiple platforms. I am excited to have played a role in building OSVR and look forward to the months and years ahead.
1
2

This article may contain affiliate links. If you click an affiliate link and buy a product we may receive a small commission which helps support the publication. More information.


Road to VR
  • vrguy

    Ben – thanks for publishing. Just as point of clarification – the diagram that appears under the ‘big picture’ heading is the current directory of open source projects. Same diagram appears on the Github site http://osvr.github.io/contributing/ and is updated from time to time.

    Additional devices that are supported by OSVR can be seen as part of VRPN:
    https://github.com/vrpn/vrpn/wiki/Supported-hardware-devices

    Our vision for OSVR is broader and includes supporting many additional devices, game engines, etc.

  • Don Gateley

    Wow! Awesome article. Thanks. Here’s hoping OSVR can drive the walled garden content systems like those for the Gear VR out of business.

    • vrguy

      Thanks, Don. We’re not trying to drive anyone out of business. We are trying to create choice for end users and opportunity for developers. Would be great to continue to grow the impressive list of partners that want to work together for “VR without borders”

  • augure

    I was dubitative at first and didn’t see a point, but after toying (or trying to) around with the DK2 and GearVR for years, I feel like OSVR will actually be the way to go in order to achieve what consumer VR is supposed to be. Oculus/Vive/PSVR are too expensive, bulky and limited, and mobileVR is too much limited, which is the reason why some people aren’t drinking the Kool Aid and are starting to warn about a scenario in which VR doesn’t take off because of artificial barriers of high-end VR and profit (no hand-tracking is conceptual non-sense for VR for example). If OSVR is accessible and modulable enough while working on most platform/engines, I think it’ll be the preferred platform to go in the coming year for VR development.