Researchers from Facebook Reality Labs and the University of Arizona published new work exploring the use of high-speed mechanical display shifting to reduce the so-called screen-door-effect (SDE) of immersive displays. SDE is caused by unlit spaces between pixels leading to the immersion-reducing appearance of a ‘screen door’ between the viewer and the virtual world. The researchers experiment with rapidly and minutely shifting the entire display to cause the display’s pixels to fill in the gaps.

SDE has been one the leading visual artifacts in modern VR headsets since the introduction of the Rift DK1 development kit in 2013. While SDE can be defeated with brute-force by employing extremely high density displays—in which the unlit spaces between pixels are too small to be seen by the naked eye—most consumer VR headsets today still exhibit SDE (with the near exception of Reverb G2), hurting immersion and visual clarity.

A real example of the screen door effect | Image courtesy Facebook Reality Labs Research

Beyond ultra high pixel density, other methods have been employed to reduce SDE. For instance, some headsets choose a smaller field of view which reduces the apparent visibility of SDE. Other headsets use a diffuser film on the display to help blend the light from the pixels into the unlit spaces between them.

Another proposal is to rapidly and minutely shift the display such that nearby pixels fill in the unlit gaps. While this might seem like it would create the appearance of a dizzying jiggling display, it’s been demonstrated with other display technologies that moving a point of light (ie: a pixel) quickly enough can create the appearance of a stable image.

SEE ALSO
'Alvo' to Bring 'CS:GO' Style Action to PSVR Next Week, Quest & Steam Versions to Follow

Researchers Jilian Nguyen, Clinton Smith, Ziv Magoz, and Jasmine Sears from the University of Arizona and Facebook Reality Labs Research explored and experimented with the idea in a paper titled Screen door effect reduction using mechanical shifting for virtual reality displays.

Rather than building a VR headset with mechanical display shifting right out of the gate, the paper’s goal was to demonstrate and quantify the efficacy of the method.

Display Actuation and Modes

 

The display actuation mechanism | Image courtesy Facebook Reality Labs Research

The researchers designed a static platform with two piezoelectric actuators which, together, shift the display in a circular motion at 120Hz—in effect, causing each pixel to trace a 10µm circle 120 times per second. The size of the circle was picked based on the distance between the display’s pixels in order to optimally fill in the unlit spaces between pixels. The researchers call this circular path ‘Non-redundancy’ mode.

They also smartly utilized a 480Hz display which allowed them to experiment with a more complex pixel shifting path which they called ‘Redundancy’ mode. This approach aimed to not only fill in the gaps between the pixels with some additional overlap, but split the displayed frame into four sub-frames which are each uniquely shifted and displayed to account for the pixel movement. This means that when a pixel shifts to a location where it would fill in the SDE gap, it uses the correct color which would be used if a pixel was located in that position in the first place.

The two pixel movement modes addressed in the paper | Image courtesy Facebook Reality Labs Research

While the paper is limited to exploring these two pixel paths, the researchers say that others could be employed based on display characteristics.

“Pixel shifting is not limited to a circular shape. Indeed, an elliptical path or even a figure-eight path could be used by controlling the amplitude of each axis’ movement. Paths can be traced in many ways to explore screen door reduction,” the researchers wrote. “For the micro OLED display, a circular path was well-suited to the square pixel and sub-pixel layouts. This path is used to balance the length of the path with the fill factor, minimizing the speed the actuators must operate at.”

The display actuation platform for experimentation | Image courtesy Facebook Reality Labs Research

With the platform built and capable of shifting the display rapidly in the desired paths, the next step was to objectively quantify the amount of SDE reduction, which proved to be difficult.

Quantitative Measurement of Mechanical SDE Reduction

The authors first sought to objectively measure where each subpixel began and ended, but found that the resolution of the camera they employed for the task was not fine enough to clearly delineate the start and end of each subpixel, let alone the spaces between them.

Another approach to quantify SDE reduction was to measure the contrast ratio of a section of the display and compare it to when the screen actuation was on vs. off. Lower contrast would imply less SDE due to moving pixels filling in the unlit spaces and creating a more solid image. While the authors maintained that this measurement isn’t necessarily a reflection of the SDE reduction as the naked eye would see it, they believe it’s a meaningful quantitative measurement.

Contrast ratio reduction in both modes at various magnification levels | Image courtesy Facebook Reality Labs Research

Qualitative Assessments of Mechanical SDE Reduction

Beyond their efforts to quantitatively measure the SDE reduction, the researchers also wanted to look qualitatively at the change. The clearest demonstration of the benefits came from looking at a natural photo with complex scenery.

Image courtesy Facebook Reality Labs Research

Here, the ‘Non-redundancy’ mode clearly reduced the SDE while apparently retaining equal sharpness. Impressively, the ‘Redundancy’ mode not only reduced SDE, but even appears to noticeably sharpen the image (note the zoomed-in sections showing details in the rear of the car).

The image sharpening of the ‘Non-redundancy’ mode is an interesting additional benefit because it actually increases the resolving power of the display without increasing the number of pixels.

Based on their experimentation the researchers also suggest a user-study approach for future investigations which could be used to quantify any SDE reduction method, whether that be mechanical shifting, diffusers, or different sub-pixel layouts and optics.

The researchers conclude:

In using mechanical shifting of pixels for screen door reduction, the dead space of the display needs to be characterized to define the path shape and shift distance required of the mechanical shifting system. With appropriate application of mechanical motion, SDE can be qualitatively reduced. A promising method of screen door visibility quantification uses natural scenes and human subjects to determine the magnification at which SDE and screen door reduction artifacts become noticeable.

– – — – –

While the brute force approach of defeating SDE with ultra high pixel density displays will likely come to fruition, a mechanical approach to SDE reduction could allow headset makers to ‘get more for less’ by boosting the effective resolution of their display while reducing SDE. This could also have knock-on bonuses to display design, as display makers would be less constrained by the need to achieve exceptionally high fill factors.

This article may contain affiliate links. If you click an affiliate link and buy a product we may receive a small commission which helps support the publication. See here for more information.


  • Jesustilloves

    I don’t see any sde on quest 2. I’ve seen it on the Pimax 5k+ but it wasn’t terrible. It’s not visible on quest 2.

    • Hymen Cholo

      I still see it. Drastically less and very tolerable but it’s there, just on the edge of invisibility. I will say I have better than 20/20 vision. We are nearly at retinal resolution but not quite.

      • Kevin White

        Just to clarify / point out, resolution and SDE are coupled in a way but aren’t the same thing. I sense significantly less SDE in the PSVR than in the Vive Pro, but no one could argue the PSVR is closer to “retinal resolution” or has higher resolution. Both these headsets use OLED, but the “RGB Stripe” configuration of sub-pixels tends to create less obvious SDE than the “Pentile” arrangement of sub-pixels. You can see those most easily by viewing cinematic material (a movie or TV show) in each, finding a very red area of a scene, and pausing on that. The PSVR presents a much more coherent red with less of a screen door interrupting the color elements than the Vive Pro does.

    • kraeuterbutter

      than you have not very good eyes…
      the SDE is very easily to be seen in the Quest2
      also on the Reverb G2 you can see it
      you dont have to search for it: white text on dark background – and i can even count the lines
      so there is still a way to go…
      4 times the pixelcount maybe at same FOV will solve it.. maybe still not enough

  • Hymen Cholo

    They use this technology in projectors to make 1080p sensors closer to 4k and it works very well.

    • Andrew Jakobs

      Not really, the pixelshift of DLP projectors works completely different. BUT as you suggest it does work for getting pixelperfect 4K (mind you, the pixelshift DLP sensors have a resolution above 1080p).

      • kraeuterbutter

        still way less than real 4k

        • Andrew Jakobs

          As it’s pixelperfect 4K (as in every pixel of the original 4k video frame is being shown) you don’t miss a single pixel….

  • Trekkie

    With one of the biggest complaints with VR being the size of the “box on your face”, wont the mechanical route simply add bulk and weight to the headset and make adoption worse?

    I would go with higher resolution microLEDs coupled with very good eye tracking.

    • Eric Nevala

      The first step in research is to establish the science and that the technique works effectively. Afterwards, it can be productized and miniaturized on the hardware engineering side.

      • Trekkie

        I was complaining about the “mechanical route”, not the gadget in its present state which undoubtedly will improve over time.

  • xyzs

    Just put a damn diffuser in front of the screen like the PSVR……

    • TechPassion

      Odyssey+

    • benz145

      Diffusers necessarily reduce effective resolution. The work here demonstrates a method for reducing SDE while boosting effective resolution.

      • piecutter2

        So, my original pixel is size 1. I spin it around to make it occupy an area that is perceptibly 4 times it’s original size, and my overall display increases by 1 horizontal and 1 vertical. I’m pretty sure I could accomplish that by adjusting the focus.
        Or perhaps I could just strap an orbital sander to my head.

        • benz145

          There’s sub-frame sampling at play which allows the moving pixels to show more information in the image than the number of static pixels would be able to.

          • piecutter2

            Yeah, I get that. It’s just looking like an overcomplicated hack that will add more weight and failure potential than it’s actual worth. It would be cheaper and simpler to manufacture higher pixel density displays. Driving them would be much simpler than having to synchronize the sampling with the orbit, and we’re nowhere near the limit of pixel density yet.

          • kraeuterbutter

            in beamers this technic is used for years now
            4k pixel-shifting
            full-hd Displays (or slightly more) which look (nearly) like 4k screen due the shifting process..
            and also it sounds more complicate than “just using a real 4k display”
            its still the way, that pixel-shifting 4k beamers costs way less than “real 4k” beamers

      • xyzs

        These complex systems add costs and weight and lower the reliability. Just increase the screen def and put a diffuser to smooth the pixel details and the job will be done right…
        These over engineered bs is not the answer to real needs. That’s it.

    • Toothlover

      calm down

    • Lucidfeuer

      I agree, polarisers and diffusers have been the credible approach from the get go but getting it right without losing quality has not been investigated further.

  • Denis

    While interesting from a research perspective, I think this is a very complicated method to solve a problem which is already solved with the type of screen used in the HP Reverb G2. High res screens with minimal space between the pixels can easily solve any screen door issue.

    There will be a demand for higher resolution screens in VR for years to come, because we aren’t close at all to the detail that desktop and smartphones screens offer (taking into account viewing distances). You wouldn’t even need to render the images at native resolution if that requires too much graphic performance. Besides, better upscaling techniques are on the rise.

    • mirak

      G2 doesn’t solve OLED blacks

      • Bob

        Colours and contrast is another area that is in urgent need of addressing in VR especially for a practical way of implementing HDR which seems to be entirely a cost issue

  • GunnyNinja

    Why should I give a damn about SDE when they won’t include my IPD?

  • Hello dear old friend ‘wobulation’! We had missed you :)

  • Andrew Jakobs

    I wonder if the extra hardware needed to shift the display doesn’t cost as much as just using higer resolution displays in the end, especially as mechanical stuff breaks down eventually.. so I really don’t see this advancing beyond the research point as it’s not very viable anymore with headsets like the G2 emerging now.

    • kraeuterbutter

      in beamers pixelshifting is used to produce form 1080p or slighty higher resolution displays a picture close to 4k
      that works very well
      and: beamer with pixelshifting still cost way less than 4k beamers..
      you get a good pixel shifting beamer at around 1500 euro
      a good 4k more like 2500-3000 euro

      • Andrew Jakobs

        Which 2500 projector has a real native 4K DLP chip? as far as I know real native 4K projector are still beyond the 3000+ euro. And then again with pixelshift (DLP versions) you don’t miss a single pixel of the 4k videoframe.

    • kraeuterbutter

      you still can see the SDE on the Reverb G2
      its even very easily be seen when you have white text on darker background
      for example the window titels in the Cliff-House-Home

      so there is still a way to go…
      don´t think that even 3k x 3k will ged rid completley of that SDE

      • Andrew Jakobs

        Yeah, you’ll probably will always see SDE if you’re really concentrating on it, even on 4K of 8K per eye, but with regular playing you won’t notice it really, not like you would on the older headsets.

  • I love research experiments like this, but let’s be honest: this most probably won’t ever see the light. Mechanical systems are much prone to break and less precise. Not to mention the fact that the system in the picture is clearly an experimental one: this is not even in the prototyping stage…

    • kraeuterbutter

      it works very well in beamers
      and – also its mechanical – beamers with 4k tilt-shifting are still much cheaper than there true 4k-beamer counterparts..

      so it still could be a rout for the next years for VR befor we realy have 8k displays (i mean real 8k displays, not the 8k pimax claims)

  • duked

    I’d prefer better screens, like G2, insted. This hack, of a constantly moving image, just brings another layer of “fake” to VR, the more you try to fool the brain, the higher the risk you will make some users VR-experience worse. I belong to those who got headache from the color wheel on single DLP projectors.

  • david vincent

    “It’s useless, the G2 already solved the problem of SDE”
    Yeah but not everybody can afford a G2 and the very high-end GPU going with it.

    • johann jensson

      Here we go again with that nonsense. You can use lower resolution target with the G2 and still have better clarity than an Index. And much less SDE, ofc.

    • Lucidfeuer

      There’s also the fact that with AI upsampling like DLSS, high resolution rendering is less of an issue.

    • kraeuterbutter

      thats simply not true
      of course you can still see SDE with the Reverb G2
      (or you have bad eyes)
      very easy to see (dont have to search for it) on white text at dark background, as example

  • MasterElwood

    4 Years to late.

    SDE was a HUGE deal with the CV1 or VIVE
    Way less of a deal with RIFT S, INDEX, Quest 1
    Almost no Deal with Quest 2 and G2

    And by the next generation it will be NO deal anymore.

    Sorry wobble-thingy – you’r to late to the party…

  • sh4dow83

    People here are missing the theoretically important thing about this tech: It would allow for more impressive visual fidelity without SDE, since one doesn’t need to throw high resolution at the headset to get rid of it.

    Of course… since almost all VR games look like crap anyway (unnecessarily, as I’ve tried out in Unity myself but… what can you do when you don’t have a few million $ lying around to make and release something that looks great yourself), I guess it doesn’t matter.

    • Bob

      No I think most understand the implications of this technology but the point isn’t about what this technology can do but rather the actual importance of it in the face of higher resolution displays which naturally solve the problem. In turn higher resolution displays can be driven by more powerful chips (tick tock cycle) and aided by foveated rendering.

      So I think it’s a matter of priorities in what needs to be researched and developed.

  • johann jensson

    Yeah, hit the nail with a rocket. We don’t want less screendoor effect through blurring, we want more resolution through redundant pixels. There are affordable 2k displays available now and 4k per eye is coming in the next 2-3 years.

    I really hope that in 3 years we’ll have the Index 2 with 2k per eye and 130° FOV.

    • johnyjazz

      If you want that FOV then you’ll be needing higher than 2k per eye!

      • johann jensson

        Then i’ll take less FOV, because resolution always trumps (is that word still allowed?) FOV.

        • oomph

          Thats why they wanted FB login to lock you out

          • johann jensson

            Can’t lock me out of something i never used. VorpX was my daily software. :-P

    • benz145

      Here, the ‘Non-redundancy’ mode clearly reduced the SDE while apparently retaining equal sharpness. Impressively, the ‘Redundancy’ mode not only reduced SDE, but even appears to noticeably sharpen the image (note the zoomed-in sections showing details in the rear of the car).

      The image sharpening of the ‘Non-redundancy’ mode is an interesting additional benefit because it actually increases the resolving power of the display without increasing the number of pixels.

      • kraeuterbutter

        pixel shifting in 4k beamers..
        its done that way for year now..

        full-hd beamers with pixel shifting look nearly like 4k
        of course not as good as 4k, but its more close to 4k than to the real nativ physical resolution
        so: it works good in beamers..
        and also fast enough (because it even works in 3d with i think 120Hz )

        4k pixelshifting beamers are still much cheaper than “real 4k” beamers

        so: with a 2.1k display like the reverb it would be maybe possible to produce the sharpness of a 3k or nearly 4k display
        and that maybe for a cheaper price

  • So basically the tech in virtualboy that made everyone vomit after 20 minutes?

    As neat as this might be conceptually, I feel like motorized components of any sort are a recipe for failure, all kinds of potential “bumped a wall, killed a motor” kind of thing and just something that isnt’ a good solution to the problem.

  • Byaaface

    Another concern is the fact a 480Hz motion might have a noticeable vibration or ‘buzz’. Not sure this could be dampened or cancelled with counter weights, but it seems like a potentially immersion breaking aspect.
    I figure for circular motion, whatever motors are used for phone or gamepad vibration could be utilized, or some piezoelectric actuator so reliability could potentially be ok. Still definitely a concern

  • FMT

    This is great news.
    There are lots of projectors that use pixel shifting to show 4k picture with 1080p chip with great results.
    This technology can produce VR screen with 4k per eye without SDE.
    It will be game changer for movies viewing.
    I would gladly replace my projector for VR with such feature.

  • Rupert Jung

    Wish they would push their varifocal optics.