NVIDIA Says New Foveated Rendering Technique is More Efficient, Virtually Unnoticeable

Researchers claim 2x-3x improved pixel shading performance with new approach to foveated rendering

28

NVIDIA Research has demonstrated a new method of foveated rendering which the company says is nearly invisible to users. Researchers behind the new approach say the method can drastically reduce rendering workload, allowing for more detailed virtual reality scenes.

The human eye only really detects significant detail in a tiny area at the center of our field of vision, thanks to a small pit in the back of the retina called the fovea. Our peripheral vision, while useful, sees things like color and movement, but very little high fidelity detail. For instance, look one inch to the left of this paragraph, and see if you can read any of the words in it. I’ll wait.

You’ve probably found that it’s nearly impossible. Foveated rendering takes advantage of this fact and supposes that, when it comes to CGI, we can get away with rendering the area outside of the fovea at lower detail, thereby being able to focus more of the available computing power in that smaller area for richer scenes.

SEE ALSO
NVIDIA Demonstrates Experimental "Zero Latency" Display Running at 1,700Hz

Foveated rendering isn’t new. It’s been around for some 20 years, according to Nvidia. But contemporary techniques resulted in objectionable artifacts like flickering or blurring which users could see too easily. So Nvidia researchers sat down to try to understand what it is that our peripheral vision is and isn’t adept at seeing.

The results of their work, soon to be published in a paper titled Perceptually-Based Foveated Virtual Reality, found that our peripheral vision is best at seeing things like color, contrast, edges, and motion. Thus, the ideal foveated rendering technique should maintain those aspects of the scene while reducing high fidelity detail. Nvidia says they used this knowledge to design a new rendering algorithm that allows for greater reduction in detail without users being able to detect it.

“When we started this project, the researchers working on it knew if foveated rendering was turned on or not. By the end, even they have to ask [whether or not foveated rendering was enabled],” said Aaron Lefohn, one of the Nvidia researchers who worked on the project.

Click to enlarge photos in the gallery above. Look at the clock highlighted by the red square. While looking at the clock, use your arrow keys to switch back and forth between the ‘blur’ foveated rendering and ‘contrast preserving’ foveated rendering. Notice the different of clarity in your peripheral vision.

The results are impressive compared to other methods. By maintaining the sorts of details that our peripheral vision is best at seeing, the rendering technique restores a feeling of clarity to the areas outside of the center of our vision. With this method, Nvidia says they’re able to reduce pixel shading performance by 2x-3x compared to a scene not using foveated rendering. Those performance savings can be used instead to make the detailed part of the image even sharper and more realistic.

SEE ALSO
NVIDIA Explains Pascal's 'Lens Matched Shading' for More Efficient VR Rendering

“There’s this insatiable demand for more pixels and more performance [in virtual reality]. The benefits we can get from foveated rendering are tremendous and growing,” said Lefohn.

Nvidia will be demonstrating the new foveated rendering method at SIGGRAPH 2016 next week.

For use in virtual reality, foveated rendering relies on eye-tracking so that the scene can be continuously updated to render sharply in the very center of the user’s vision, with the foveated rendering only kicking in on the peripheral. Nvidia partnered with SMI which supplied the researchers with a VR headset with inbuilt eye-tracking tech capable of accurately tracking the eye’s gaze direction 250 times per second.

“You can’t outrun [the eye tracking] with your eyes. It’s fast enough and accurate enough to give us a foundation to build [foveated rendering technologies] on,” said Lefohn.

SMI-DK2-Eye-tracked (2)
A VR headset equipped with SMI’s eye-tracking tech

So we won’t be seeing foveated rendering in consumer virtual reality until VR headsets adopt eye-tracking technology. Japan-based FOVE raised $11 million following a successful Kickstarter to develop a consumer VR headset with built-in eye tracking. Major headsets like the Rift, PSVR, and Vive have yet to ship such technology, but many suspect it may be in the works for future generations of the headsets.

This article may contain affiliate links. If you click an affiliate link and buy a product we may receive a small commission which helps support the publication. See here for more information.


Ben is the world's most senior professional analyst solely dedicated to the XR industry, having founded Road to VR in 2011—a year before the Oculus Kickstarter sparked a resurgence that led to the modern XR landscape. He has authored more than 3,000 articles chronicling the evolution of the XR industry over more than a decade. With that unique perspective, Ben has been consistently recognized as one of the most influential voices in XR, giving keynotes and joining panel and podcast discussions at key industry events. He is a self-described "journalist and analyst, not evangelist."
  • David Herrington

    What about using oversampling only in the foveated rendered center area to increase perceived quality? I understand this is a brute force method and the real answer is better resolution and optics but it would get us higher quality in the meantime.

    • Jachim Soyer

      Honestly, Nvidia doesn’t want to talk about it as it doesn’t help their bottom line much, but this’ll get VR headsets onto PC’s that might not be able to run em’. I love the idea.

      • Andrew Jakobs

        No this might help get higher resolution into vr headsets without having to have a quad sli titanX setup needed to render it.. there is no GPU which is capable of rendering 4k displays (per eye) at the needed framerate in full glory detailed graphics..

    • beestee

      To me it seems that this technology is right on pace since it is just not possible to gain any advantages from any form of foveated rendering until eye-tracking is available in consumer hardware.

      There is no advantage to be had in regards to foveated rendering with the consumer HMDs available today.

      • David Herrington

        Oh, yeah, I mean obviously we would need to add eye-tracking into consumer hardware. But that’s already technology we already have available today and can put in. I say that version 2.0 needs to bring better optics, better resolution, and eye-tracking to really boost us into the next level!!!

  • Jon

    I’m just back from a classic English pub having had one or two (six or seven) too many so my sight is slightly blurry where ever I look but this is the the simplest and most exciting explanation I’ve read for foveated rendering (and eye tracking) yet… roll on 2017 / 2018… the exciting times continue to head our way…

  • It’s fun that people are looking so far forward with VR technology, I really hope all of these fancy eye-tracking, performance enhancing things happen… but…. let’s put the rabid fanboy-ing aside for a second and keep in mind that, right now, there really aren’t any killer-apps for VR, and mass adoption of VR is still more of a hope then a reality.

    I love VR, and I’ve spent thousands on it over my life. This is the closest to mass adoption we’ve ever seen it. But my VIVE isn’t getting a ton of use, other then my own Unreal Engine development. I haven’t used my GearVR hardly at all. It’s not hardware we’re lacking right now, it’s better software.

    • Buddydudeguy

      ” over your life”?? Real VR only just came out this year.

      • It’s cute that you think of the recent movement in VR as “Real VR”. VR has been around since the 70’s when Mainframes powered the visuals. This is a large push into the consumer market of something that’s existed for DECADES, and not even the first big push, but the second.

        And, even with this push of new technology, we still have no full body tactical feedback suits, omni-directional treadmills are still a dream, and there aren’t even tactical feedback gloves available for purchase yet!

        “REAL VR” should have a neural interface like Johnny Mnemonic or the Matrix. People 10 years from now will be talking about how they have “REAL VR”. People 100 years from now will be talking about how they have “REAL VR”. You’ll be long dead from old age before people stop talking how how much of a joke all other VR was before their time.

        Dig in and get comfortable, kid. I’ve been into VR for a long time, and I can tell you, it’s not a race, it’s a marathon. What you see now will be a joke in 5 years. Get used to it.

        • Buddydudeguy

          What’s cute is you think that crap you’re referring to was VR. We all know who the “kid” is, it’s the tool using “kid” as a jab.
          VR has not existed for “decades”. How can you be into something ” for a long time” when it just came out a few months ago? You’re delusional lol

          There’s a reason it was never taken seriously till now. We broke the first important barriers and that’s frame rate, tracking latency and acceptable graphics. The crap you maintain as VR was non-viable in any real sense.

          • Oh man. No need to take anything you say seriously anymore. You seriously said VR came out a few months ago. OMG, so funny. Please, say something else. Let’s see how far that foot fits in your mouth.

          • Buddydudeguy

            You’re an idiot. You don’t get it. Anything trying to be VR before the Vive and Rift failed for numerous reasons namely frame rate, tracking latency and FOV. Get it got it good? Not the brightest bulb in the bunch….

          • LoL! You’re funny, kid. I suppose there’s no reason to tell anything, since you’re certain you already know everything (lol), but for anyone else that’s curious:

            —- For the Adults still reading—-
            Industrial and scientific VR has been sold since the early 80’s, with experimental stuff even before that. Systems like “Cave” have been featured in the news, in business and university settings, for over 20 years. The company “Virtuality” marketed a successful line of VR arcade machines in the late 80’s, which included the VR classic “Dactyl Nightmare”. The VFX1 and CyberMaxx allowed people with deep pockets to play Quake in VR in the late 90’s.

            Every era of VR promised big, had big support, made headlines… and sorta petered out. But every step of the way the price was halved and the performance was quadrupled. We have very good headsets now, with high polygon count, better rez, and lower latency then ever before.

            But, even now, people are talking about “Screen Door Effects”, Omni-Directional treadmills that still aren’t out, tactical feedback that is still experimental, and 4k to 8k headsets that just aren’t possible yet. Despite this demand for even more, overly enthusiastic VR fans are talking about how revolution is NOW. I certainly hope so, but I’ve heard this before. Each time it has failed because the demands of people who “Wait and See” vastly outstretch the technology of the time, and all of that imagined demand never manifested. Since enthusiast sales alone can’t support most corporate models, companies go under, and VR goes into hibernation for another decade.

            What we have now is amazing, but it still isn’t “Perfect”. I would hope that enthusiast sales will be enough to keep a stable market for VR tech, allowing VR to prosper, even if it doesn’t hit the high expectations of the mainstream and flops. At least then we won’t have to return to the days of cobbled together tech and pricey one-off headsets with no software support.
            ——- End —–

            As for you, kid, you might want to quit before you make a bigger ass out of yourself. I know, you won’t. Kids like you always need the last word. Feel free to take it. I think I’ve said enough. I’m deleting all further notifications on this thread.

          • Buddydudeguy

            Jesus there he goes again. Blocked to avoid notifications from the idiot who thinks VR was around in the 70’s.

    • Jordy

      You’re wrong, we need both, better hardware and software. You can still see pixels in Rift CV1 or Vive, a better screen and resolution will make a huge difference, but we also need better graphic cards.

  • leadingones

    If you are looking to get into development for virtual reality check out
    leadingones.com
    We have free tutorials covering unity and unreal for both the Vive and the Rift!

  • ZenInsight

    This seems like the answer to rendering issues at higher rez, but when will eye tracking be inside headsets? By the time that happens 3-5 years, the level of graphics that could be displayed with a single decent GPU combined with this could turn VR fairly photo realistic. I wonder how much improvement in frame rates this manages. They says 2x-3x times, meaning something that ran at 30 frames would run at 60-90 frames…but who believes the GPU designers. Real world stats or it didn’t happen.

    • InquisitorDavid

      They said 2-3x *pixel shading* performance. There are other parts of the graphics rendering pipeline that don’t get an uplift.

      But what that does is give developers a little extra rendering power to use for increasing the detail in the focus area. Quality will always be a moving target, so squeezing every ounce of performance from any part of the pipeline is welcome.

      As a recent example, Shadow Warrior 2 is the first game to utilize Multi-Res Shading, which is an implementation of some of the technologies used in Foveated Rendering. With a GTX1080 @ 4K, it gives up to a 30% frame boost, allowing it to cross the 60fps average barrier.

      With VR (that uses 2 eyes), you can save even more processing power by using LMS (which renders the lens-warped parts of an image at a lower resolution), Eye Tracking + MRS, one can probably get 40% more actual total performance.

  • Freedom

    This is a form of Depth of Field in which we would render the scene and then blur the hell out of it. We wasted a bunch of calculations to render that pixel only to blur it. This technique is not just for VR but could be extended to all rendering. Sure if you focus your eyes on the stuff you are not suppose to focus on it would appear blurry but film makers and games have been using this technique to draw your focus where they want it for years. Nvidia has just decided to do it more efficiently. The more efficient we render the better the graphics we can produce. Simple but elegant the best solutions to anything.

  • Buddydudeguy

    I”M retarded. I’m not the one maintaining there was VR in the 1860’s. No, you misunderstand me. I’m saying they tried to do VR over the years, but the tech wasn’t there. YOU chose to hang on to words literally. No…there was no VR in teh 90’s and there sure as shit wasn’t VR in teh 1860’s. Who’s retarded again? zero percieved latency was NOT availabe tech in the 90’s or the 1860’s. Now kindly fuck off. By the way, nice necro moron. A year later, OMG SOMEONE ON THE INTERNET ! I NEED TO PLAY WORD GAMES WITH HIM AND APPEAR SMART! get a life.

    • DJHeroMasta

      “No…there was no VR in the 90’s and there sure as shit wasn’t VR in teh 1860’s.” Give the guy a quote and he’s still in denial….smh.

      • Buddydudeguy

        You can’t wrap your head around the fact that wasn’t viable, zero latency VR.

        • DJHeroMasta

          What current consumer headset has Zero latency? I’ll wait. (–_–)

          • Buddydudeguy

            the frack are you talking about??? The Rift and Vive have zero perceived latency. VR in the 90’s? Not so much. Stop trying to one up me with your retarded arguments. You fail. Hard.

          • DJHeroMasta

            They both have “near” zero latency. Their trackers poll at 120hz. That’s an 8.3ms delay.

          • Buddydudeguy

            You’re not done eh?

            There is zero PERCEIVED latency.

          • DJHeroMasta

            Define Perceived.

          • Buddydudeguy

            wooooow this guy. 90fps with a Rift or Vive is zero perceived latency, and you know it.