Reducing Latency is Becoming Increasingly Complex

Trends

Presence in VR requires low latency, and reducing latency is not easy. Low latency is also not the result of one single technique. Instead, many methods work together to achieve the desired result. ’Asynchronous time warp’ modifies the image just before sending it to the display. ’Predictive tracking’ lowers perceived latency by estimating future orientation. ’Direct mode’ bypasses the operating system. ‘Foveated rendering’ reduces render complexity by understanding eye position. ’Render masking’ removes pixels from hidden areas in the image.

nvidia-17000hz-display-zeo-latency-2
See Also: NVIDIA Demonstrates Experimental “Zero Latency” Display Running at 1,700Hz

If this sounds complex, it is just the beginning. One needs to measure optical distortion and correct it in real-time. Frame rates continue to increase, thus lowering the available time to render a frame. Engines can optimize rendering by using similarities between the left- and right-eye images. Techniques that used to be exotic are now becoming mainstream.

Only a handful of companies have the money and people to master all these techniques. Most other organizations prefer to focus on their core competencies. What should they do

OSVR implications

A key goal of OSVR is to ‘make hard things easy without making easy things hard’. The OSVR ’Render Manager’ exemplifies this. OSVR makes these latency-reduction methods available to everyone. We work with graphics vendors to achieve direct mode through their API. We work with game engines to provide native integration of OSVR into their code.

I expect the OSVR community to continue to keep track of the state of the art, and improve the code-base. Developers using OSVR can focus away from the plumbing of rendering. OSVR will continue to allow developers to focus on great experiences.

The Peripherals Are Coming

Trends

A PC is useful with a mouse and keyboard. Likewise, a VR headset is useful with a head tracker. A PC is better when adding a printer, a high-quality microphone and a scanner. A headset is better with an eye tracker, a hand controller and a haptic device. VR peripherals increase immersion and bring more senses into play.

In a PC environment, there are many ways to achieve the same task. You select an option using the mouse, the keyboard, by touching the screen, or even with your voice. In VR, you can do this with a hand gesture, with a head nod or by pressing a button. Applications want to focus on what you want to do rather than how you do it.

More peripherals mean more configurations. If you are in a car racing experience, you’d love to use a rumble chair if you have it. Even though rumble chairs are not commonplace, there are several types of them. Applications need to be able to sense what peripherals are available and make use of them.

Even a fundamental capability like tracking will have many variants. Maybe you have a wireless headset that allows you to roam around. Maybe you sit in front of a desk with limited space. Maybe you have room to reach forward with your hands. Maybe you are on a train and can’t do so. Applications can’t assume just one configuration.

OSVR Implications

OSVR embeds Virtual Reality Peripheral Network (VRPN), an established open-source library. Supporting many devices and focusing on the what, not the how is in our DNA.
I expect OSVR to continue to improve its support for new devices. We might need to enhance the generic eye tracker interface as eye trackers become more common. We will need to look for common characteristics of haptics devices. We might even be able to standardize how vendors specify optical distortion.

This is a community effort, not handed down from some elder council in an imperial palace. I would love to see working groups formed to address areas of common interest.

Turning Data into Information

Trends

A stream of XYZ hand coordinates is useful. Knowing that this stream represents a figure-eight is more useful. Smart software can turn data into higher-level information. Augmented reality tools detect objects in video feeds. Eye tracking software converts eye images into gaze direction. Hand tracking software converts hand position into gestures.

Analyzing real-time data gets us closer to understanding emotion and intent. In turn, applications that make use of this information can become more compelling. A game can use gaze direction to improve the quality of interaction with a virtual character. Monitoring body vitals can help achieve the desired level of relaxation or excitement.

As users experience this enhanced interaction, they will demand more of it.

OSVR Implications

Desktop applications don’t have code to detect a mouse double-click. They rely on the operating system to convert mouse data into the double-click event. OSVR needs to provide applications with both low-level data and high-level information.

In ‘OSVR speak’, an ‘analysis plugin’ is the software that converts data into information. While early OSVR work focused on lower-level tasks, several analysis plugins are already available. For example, DAQRI integrated a plugin that detects objects in a video stream.

I expect many more plugins will become available. The open OSVR architecture opens plugin development to everyone. If you are an eye tracking expert, you can add an eye tracking plugin. If you have code that detects gestures, it is easy to connect it to OSVR. One might also expect a plugin marketplace, like an asset store, to help find and deploy plugins.

Continue Reading on Page 3 >>

1
2
3

This article may contain affiliate links. If you click an affiliate link and buy a product we may receive a small commission which helps support the publication. See here for more information.


  • Get Schwifty!

    One thing for open development to remember – polish it up!!! Too many times consumers have an adverse reaction to Open Source development because the end result works (often barely), but feels janky like it’s half done. The idea of Open Source is great, it’s getting the discipline out of folks to do the last 10% on things that often cripples the effort.

    • Bryan Ischo

      The hard part about that is that the last 10% is usually a lot of effort and most of it is drudgery, and people who are not paid for their time would rather do the fun bits.

      It’s all open source though, anyone who doesn’t like “janky half-done” projects can put their time and effort towards improving the project.

  • Bryan Ischo

    I’m not going to click your ad link.

  • dev

    If you “would like to see OSVR expand” then provide some incentive to devs rather than insults to devs from your Razor-assed proprietary partners!

  • Jan Herca

    Most miportant in a VR system is tracking system because that is going to limit the rest of the devices (HMD, controllers, etc).
    Because that a VR system must start in the tracking system. Oculus has Constellation. Valve has Lighthouse. They are two opposed systems in the way the operate, the aren’t compatibles.
    And because that there cannot be a mix-and-match system.
    OSVR is just another way to sell a VR system using the word “Open” as a marketing “buzzword”, but as soon as it standarize a tracking system for it will be as close as any other VR system.

    • VRguy

      OSVR supports Oculus tracking, Lighthouse tracking and over 100 (yes, 100) different tracking systems. See http://osvr.github.io/compatibility/ for a full list. OSVR is justified in calling itself ‘open’ because it is open-source so anyone can add, change or modify the tracking cost or almost any other piece of the code.

      • Steve

        I have rewritten some of those drivers because they were too buggy, and I am not putting that work out there for free!

        • VRguy

          We’d appreciate it if you put out some of your fixes, or at least file a bug report. By working together, the community can make progress faster.

          • Jobz

            Oh yeah, to make things easier for your Razor tongue buddies to make sick games spread further!

  • Boshir Moslm

    Hi,

    I have to say it’s awesome. Virtual
    Reality One of the first things in weeks 2 in 1 VR
    Case that have impressed me much with VR. It is a protective case to protect my
    smartphones for drop and scratch. Enjoy the first time feeling of putting VR
    into my pocket . Thanks!https://www.iwearvr.net/products/zero-vr-2-in-1-case-virtual-reality-case-cell-phone-cover