Facebook has developed an AI-assisted method for supersampling real-time rendered content, something that could become an integral addition to games coming to future generations of high-resolution VR headsets.

There’s an ongoing arms race between display technology and GPUs, and adding VR into the mix only underscores the disparity. It’s not so much a question of putting higher-resolution displays in VR headsets; those panels are out there, and there’s a reason many manufacturers aren’t throwing in the latest and greatest in their headsets. It’s really more about hitting a smart balance between the display resolution and the end user’s ability to adequately render that VR content and have it look good. That’s the basics anyway.

That’s why Facebook is researching AI-assisted supersampling in a recently published paper, dubbed ‘Neural Supersampling for Real-time Rendering’. Using neural networks, Facebook researchers have developed a system capable of inputting low-resolution images and obtaining high-resolution output suitable for real-time rendering. This, they say, restores sharp details while saving computational overhead.

 

Researchers claim the approach is “the first learned supersampling method that achieves significant 16x supersampling of rendered content with high spatial and temporal fidelity, outperforming prior work by a large margin.”

From the paper:

“To reduce the rendering cost for high-resolution displays, our method works from an input image that has 16 times fewer pixels than the desired output. For example, if the target display has a resolution of 3840×2160, then our network starts with a 960×540 input image rendered by game engines, and upsamples it to the target display resolution as a post-process in real-time.”

If all of this sounds familiar, that’s because it’s a similar concept to Nvidia’s Deep Learning Super Sampling (DLSS), which is currently only available on its RTX GPUs.

Facebook researchers say that methods like DLSS however either introduces “obvious visual artifacts into the upsampled images, especially at upsampling ratios higher than 2 × 2, or rely on proprietary technologies and/or hardware that may be unavailable on all platforms.”

SEE ALSO
Google Takes a Step Closer to Making Volumetric VR Video Streaming a Thing

Moreover, Facebook’s Neural Supersampling approach is said to be easily integrated into modern game engines, requires no special hardware or software such as proprietary drivers (like with DLSS). It’s also designed to be compatible with a wider array of software platforms, acceleration hardware and displays.

Oculus Varifocal Prototypes, Image courtesy Oculus

It’s admittedly a difficult problem to address, and Facebook says there still needs to be more brains working on the issue to bring it to fruition, so we may not see it for some time, at least not direct from Facebook.

“This work points toward a future for high-resolution VR that isn’t just about the displays, but also the algorithms required to practically drive them,” the researchers conclude.

This, combined with a number of technologies such as foveated rendering, may bring the generational leap developers are looking for, and truly give VR games an undeniable edge in the visual department. VR games are often maligned for their lower-quality graphics, something will hopefully soon be an anachronism as we approach an era of photorealistic virtual reality.

– – — – –

If you’re looking for a deeper dive into Facebook’s machine-learning supersampling method, check out the full paper here, which is slated to be presented at SIGGRAPH 2020 this summer.

Alternative Text

This article may contain affiliate links. If you click an affiliate link and buy a product we may receive a small commission which helps support the publication. More information.


Well before the first modern XR products hit the market, Scott recognized the potential of the technology and set out to understand and document its growth. He has been professionally reporting on the space for nearly a decade as Editor at Road to VR, authoring more than 3,500 articles on the topic. Scott brings that seasoned insight to his reporting from major industry events across the globe.
  • mfx

    The amazing result looks almost too good to be true. Like the NY text. How are all the font extra details found ? Maybe it’s using more data than the low quality picture as source.
    If this is all true, I am very impressed by Facebook rnd! Especially with the platform agnostic nature of the algorithm.

    • Kevin White

      That’s a good question, especially concerning the city names signage on the wall. It reminds me of this recent post:

      https://aiweirdness.com/post/622002033086578688/depixellation-or-hallucination

      If you open the example GFYCAT image in a separate window, click the SD/HD button, and take a screenshot of the “before” and “after” ( https://imgur.com/a/6me3tmw ), it just doesn’t seem possible an AI could take nothing but the image and “enhance” it to that degree without some additional data or assistance. I’m skeptical but I’ll read up more on what they’re doing.

      • Bob

        “without some additional data or assistance”

        That’s why it’s called a neural network ;)

      • Adrian Meredith

        they use temporal information which means it can effectively infer almost double the resolution based on what pixels were there in the previous frame. This is what the motion vectors are for. Its the same trick google uses in its pixel phone cameras to do super zoom and hdr.

      • Zbyszek

        You have to realize that neural networks are not like any other algorithms. Neural networks in a way interpret what they get on the input and provide an output which is somewhat related to the input, but no necesarilly in a direct way. For example, a neural network can see a picture of a cat and say “it’s a cat”. The neural network interpreted the picture and gave you a result which you desired. Or it can see a word “cow” and it can give you an image of a cow in return. It can also very well read a blurry text, look at the general shape of the font and then produce a completely new text with this font. It doesn’t have to reproduce pixels just on the basis on other pixels, it’s not this kind of direct algorithm.
        Of course what I wrote is more of a highlevel look on the issue, in practice you provide a neural network with input, then it produces an output and then you say if it’s a proper output and then repeat millions of times. There is no explicit reasoning like this is a text and so on, but the key takeaway here is that neural networks do come with additional information, all the millions of previously solved cases concentraded inside the neural network.

  • kontis

    It’s not so much a question of putting higher-resolution displays in VR headsets; those panels are out there,

    This is FALSE. VR headsets are usually at the max or close to the max of Pixel Per Inch specs for the type of display they use (especially true for OLEDs; in case of LCD sometimes other specs are prioritized). Nothing better is available to buy in the world. Low quantity lab samples and microdisplays (too small for VR) don’t count.

    Why is RoadToVR spreading this popular, false myth created by people on Reddit? Just because it sounds reasonable does NOT mean it’s true.

    VR industry is based on the best of display fabrication created for smartphones. Samsung had plans for 4K OLED Galaxy Note in 2015, but they realized pushing resolutions on smartphones that far makes no sense, so they halted these risky multi-billion-dollar investments. THIS is the only true reason we still don’t have 4K per eye OLED VR. Everything else, like GPU power, is secondary.

    Mobile GPU in Oculus Quests pushes resolutions quite close to what PCVR users have with a 20x more powerful GPU. This ridiculous “displays exists, but we don’t have power” myth makes zero sense. No, these super high resolution displays do NOT exist.

    • Blaexe

      That statement could absolutely refer to the panels the HP Reverb uses. And yes – these are there. But almost no company decides to use them.

      • kontis

        If you want 120-144hz or deep blacks of OLEDs you cannot achieve this kind of resolution in a VR headsets with currently available screens. Also these screens had more issues (redness, mura in Reverb 1), so they are just now becoming good.

        When Vive and Rift CV1 released they had the highest mass manufactured PPI OLEDs ever made in history of this planet. But despite thath fact there was a popular myth “yeah it’s low res because GPUs are too weak” spreading like Covid-19, so I’m disappointed seeing RTVR, which is a good source, spreading this lie further.

        Again, the most expensive enterprise headsets confirm it’s a myth.

        • Blaexe

          Now you’re shifting the goal post to refresh rate, which the author didn’t talk about at all. And I wasn’t specifically talking about the G1. The G2 seems way better.

          You really shouldn’t read so much into something that’s not there.

          • kontis

            I’m not shifting the goal post. The author was literally claiming that the state of resolution in VR industry is caused by insufficient rendering power which is simply a 100% LIE, created purely from reasoning and not sources and facts.

            That’s the problem.

          • Blaexe

            The author is saying that there are higher res panels available and many companies do not use them.

            I gave you an example of exactly that. A higher res panel that is not widely used. Refresh rate doesn’t matter, it’s about resolution.

          • Ad

            He means that LCDs can only push one parameter beyond the current high end by sacrificing others. So when you say that the G2 exists, he’s saying that it took a lot of sacrifices to get there. So the limiting factor is still the display tech.

          • Bob

            If you mean contrast? Sure.

            Everything else? Not really.

        • Adrian Meredith

          for a start its a myth that oleds have deep blacks. Low persistence oleds don’t turn off their pixels like normal ones do so its still a bit greyish. much better than lcd for sure but certainly not what I call “deep”

    • Bob

      So what is your point exactly?

      The reason resolution isn’t there is because the costs are far too high for a financially viable product for the end-user?

      These “mythical” panels must be mass produced in order to break the myth?

      Performance/processing power is also an issue and always has been for the end-user. The infrastructure isn’t there, for the average end-user and not the 1%, to affordably access these sort of ultra high resolutions. So it’s a double whammy.

      This infrastructure is both software and hardware because it takes into account the software engineering necessary to develop a robust foveated rendering and upscaling solution to solve partially the hardware problem (GPU processing power). And the hardware to support that.

      Once you have those things, then it’s just a matter of waiting for the costs to come down for these “mythical” displays you’re ranting on about.

      • Ad

        I think he means that smartphones are the main industry pushing display tech, and there effectively are no better displays because smartphones don’t need them. They’re not “out there” because they’re not being made and VR isn’t the kind of massive market needed to build them efficiently. These things are developed as they’re sold, so they’re not on a shelf somewhere and can’t be.

    • nejihiashi88

      ever heard of 4k displays phones from sony? yes we have the displays but we don’t have enough gpu power , oculus quest have 72 hz resolution at the middle with lower resolution elsewhere, also they are massively downgrading the graphics in quest with crazy optimisations, the gpu is really behind that is why nvidia came with dlss and cant keep raytracing at high fps.

      • Ad

        the gpu is really behind that is why nvidia came with dlss and cant keep raytracing at high fps.

        DLSS was made to make ray tracing viable, which is different than resolution and not a display tech issue. It also helps make their low end cards more viable. The Quest also is not a great comparison considering it’s using an old phone chip.

        • Mradr

          DLSS wasn’t made for RT – it was made for two different reasons – one as a SS replacement for the higher end scope and two for the lower end scope use AI to upscale from a lower res input to a higher output in a smart way.

          • Ad

            I remember they using them together pretty aggressively to make Ray Tracing viable. Cyberpunk 77 basically requires it if you want to enable ray tracing.

      • DanDei

        If you mean the 4k display of the Sony Xperia 1 II, that has a refresh rate limit of 60hz. That is not enough for proper high end VR.

        • nejihiashi88

          well that’s the point they won’t do it because it either will lower the battery performance or will cause heat issues, and it’s already affecting the battery with high refresh rate phones, also there is no gpu that can do 4k 120hz let alone do it on a mobile phone lol

      • mepy

        Exactly the problem was the GPU’s, but there are enough good enough GPU’s to support better resolution than 2880×1440 now. Also Nvidia Ampere expected in September with 50% better performance.

    • mepy

      What you are saying is not correct. There have been OLED 1,443 ppi PPI prototype displays since May 2018 made by LG and Google. While Japan Display has 1001 PPI also since May 2018. Mobile screens are no more than 4K because they don’t need to be, we can’t see more detail than 4K on a phone. It’s also not the same screens that are used in VR and on phones.

    • Mradr

      Your whole argument is false though too. We do have headsets that ran 4k by 4k – eye tracking, etc etc – the problem is that they >cost< somewhere close to house. Not even enterprise really wants it. So you are not fighting the best of the best display issue – you are fighting cost and availability issue.

  • Rudl Za Vedno

    Sebastian from MRTV has received HP Reverb G2. He says it has the best PQ available to consumer. Even better than 8KX. Here’s Half-Life: Alyx looks Like through the lenses with G2:
    https://www.youtube.com/watch?v=vWhOA-PJAek

    • Andrew Jakobs

      With a direct comparison of another headset I think it’s a bit difficult to really tell how it really looks. But it is promising, shame that the rest of the headset probably isn’t on par..

      • Did you mean without a direct comparison? (I agree, btw, just asking)

        • Bob

          Yes I believe that’s what he meant.

        • Andrew Jakobs

          Yeah, it’s what I meant. I sure looks good on the video.

      • Ad

        You could compare it to the rest of his through the lens videos. And what do you mean? The Reverb G2 is the best headset on all fronts besides the Index.

        • Adrian Meredith

          its controllers don’t have capacitive buttons for a start let alone basic finger tracking ala index. Also its not wireless.

          • Ad

            1. I said it’s not as good as the index, I’m comparing to Rift S.
            2. No headset is wireless unless you’re including Vive Wireless which is fair, or Quest with streaming which is not.

          • Zbyszek

            Even compared to Rift S the controllers seem to be a bit lacking. Apart from that, G2 looks really good.

          • Bob

            “Even compared to Rift S the controllers seem to be a bit lacking.”

            If a bit lacking you mean the lack of capacitive sensors then sure otherwise, by and large, they are mostly the same. Of course taking into consideration the slightly varied form factor.

            The capacitive sensors are hardly ever used in most use-cases so in my opinion they won’t be missed.

            If a new VR user wanted an affordable high-end VR system, this G2 seems to fit this category very well.

          • Mradr

            I dont think wireless is that big important when you really break it down. You can still have fun with or without cable.

            I do think capacities buttons around the Touch range is best – while steams controllers are a bit over done – Even none capacities touch buttons can still work with Alex.

      • Oliver Cook

        “, shame that the rest of the headset probably isn’t on par..” How do you know? Have you tried it??

    • mfx

      Out of subject

    • impurekind

      Looks great. Can’t wait until this kind of quality and beyond is standard for all VR headsets.

  • Ad

    This is called DLSS and it already exists, and NVIDIA has made a couple of VR centric versions on top of that, although I expect DLSS to just get extended to VR. Maybe don’t act like facebook is the first to the party when they’re like 2 years after it’s invented.

    Somewhat more interesting is ASW, which reduces artifacting a lot because it’s depth aware, and maybe with some AI training the way NVIDIA does it, we could get motion smoothing in VR that feels only slightly less natural than native 90, 120, 144.

    • ender707

      This is from the article;

      “If all of this sounds familiar, that’s because it’s a similar concept to Nvidia’s Deep Learning Super Sampling (DLSS), which is currently only available on its RTX GPUs.

      Facebook researchers say that methods like DLSS however either introduces “obvious visual artifacts into the upsampled images, especially at upsampling ratios higher than 2 × 2, or rely on proprietary technologies and/or hardware that may be unavailable on all platforms.”

      • Ad

        1. They haven’t show that this doesn’t especially on an Soc.
        2. This is for facebook so it is available on all platforms how?

        • Zbyszek

          The person you reply to pointed to the simple fact, that you didn’t read the article before commenting which is obvious, because you said “This is called DLSS and it already exists” and the article says it also in plain words. Unless you adress what he said in his post, there is no point for him to answer you.

          Ad 2. DLSS requires NVIDIA proprietary tech/drivers, what Facebook is doing is more open and is intended to be widely used in the industry. It’s not as much for Facebook, but rather by Facebook (and propbably others). It’s all in the article.

      • Adrian Meredith

        tbh dlss 2.0 looks much better than this but yes as they stated it relies on proprietary technology. Lets hope the XR2 can handle this (it does claim to have 10x ai performance so theres hope)

      • CursingLlama

        Problem is from what I’ve read this is pretty much identical tech to DLSS. They are both using neural net AI systems to upscale content. The reason DLSS is limited to Nvidia GPUs being they run the neural net on the tensor cores of the RTX cards. There are also rumors that a DLSS 3.0 is on it’s way that will eliminate the need for per game training and work automatically with any game that already implements TAA.

  • Sven Viking

    ENHANCE!

  • notRobot2

    We need goggles not boxes on ur face

    • Adrian Meredith

      yes and?? whats that got to do with anything? There are a 1000 problems to solve this is one of them. FB literally showed off glasses size displays earlier in the week

    • gothicvillas

      And don’t forget we also need flying cars and unicorn pets

    • indi01

      they will be goggles.

  • MadMax1998

    “Sixteen times the detail!” ;)

    • david vincent

      Add AI SSAA to Fallout 76 and you have 16 * 16 = 256 times the detail
      MINDBLWO

  • david vincent

    It reminds me those Hollywood flicks where they can magically recreate details from very pixelated images. Except there is no magic here, only extra depth and motion data. Very impressive tho ! If really any GPU (ATI, Nvidia, mobile GPU…) can benefit from it, we may have a little revolution here (and not only for VR).

  • Very interesting research, facebook is also using AI to reconstruct the part of the images rendered at very low quality in foveated rendering… they are putting AI at work for VR!

  • Dave

    Hey guys, but confused deep learning for Oculus VR was discussed 2 years ago at OC5. Is this a rehash of that? https://youtu.be/uUdZFge6ldI?t=979

  • impurekind

    Hopefully everything they’re developing is going to lead to one pretty great gen 2 Oculus Quest. . . .

  • VR Geek

    Wow. The AI was able to get the name London and even Berlin off that low res mess? I am struggling to understand how it could have done this unless it already recognized it it from its learning photo data set. Wow I am skeptical however, especially when it makes errors which will likely feel weird.

  • mfx

    I we picture a 4k per eye headset with eye tracking in a near (finger crossed) future,
    foveated rendering + this supersampling can be used to optimize the rendering cost a lot by computing 4 viewports, with 2 large FOVs and 2 narrow FOVs:

    -the background large FOV 4k image can be generated from a low quality 1k image + 16x supersampling,
    -the gaze narrow FOV could be something like 1k + 4x supersampling for extra sharpness,
    and with this have a vr experience similar to 8k*4k for the computing cost of 2k*2k only : VR has a bright future :)