Oculus CTO John Carmack has developed a new approach to VR video playback which makes it possible to play 5K 3D 360 video on headsets like Oculus Go and Gear VR. The results, as shown in the recent re-release of Henry, are very impressive. In a new blog post, Carmack breaks down how the player works and offers up a sample for developers interested in using the technique.

If you download the latest version of Henry on Go or Gear VR, you’ll see a scene with quality which looks nearly as sharp as real-time rendering, but with much greater detail than could be rendered on a mobile VR headset, thanks to the fact that you’re actually looking at pre-rendered footage. Not only is the detail impressive, but the 60 FPS playback makes things look smooth, and it’s 3D to boot.

‘Henry’ was the first production from the now defunct Oculus Story Studios. It was original released at the launch of the Oculus Rift in 2016 and was rendered in real-time. The short film was eventually brought to Gear VR and Oculus Go as a 360 video, and has now been re-released on those platforms with Carmack’s new playback technology. | Image courtesy Oculus

This new bar in quality is thanks to a novel technique to playing back 360 video which aims for maximum fidelity with minimal sacrifice elsewhere. Developed by Oculus CTO John Carmack, the new player eschews common projections like equirectangular in favor of a novel scheme which simultaneously decodes four video streams: a ‘base layer’, and three high resolution strips which shift as the viewer looks around the scene. Carmack explains in a post on the Oculus blog:

As a base layer, the entire 5k master is resampled to a 2k x 2k version that is always available, so even if you spin around 180 degrees in a split second, there will always be something sensible to see. If you look at the ceiling or floor, this version is all that you will see.

The key center regions are cropped directly out of the 5k x 5k master, pixel for pixel with no resampling. These are split into ten independent video files, a 512 x 1024 pixel column from each eye, stacked and transposed to make 2048 x 512 video strips.

The total file size is roughly what a traditional 5k x 5k encoding would be because skipping the poles balances out the extra gop keyframes and extra low detail copy of the middle.

He notes that this approach effectively serves up a central band of high resolution footage in a circle around the user, but sacrifices quality toward the top and bottom of the image, which will make the approach less suitable for certain video production that make heavy use of that space. Carmack explains why he decided to make this particular sacrifice compared to something else (like reducing frame rate, going from 360 to 180, or dopping 3D):

The key compromise in the current scheme is to accept lower resolution at the top and bottom of the sphere. I justify this in several ways:

  • Most people don’t look very far up or down when watching 360 video. Of course, you can follow this a little farther and say they don’t look left and right all that much either, and you wind up with the strong argument for 180 video, but 360 is still an incremental value.
  • The down direction is often blacked out to hide the camera rig.
  • For 3D 360 videos, directors are strongly urged to keep people from looking up or down, because the stereo 3D effect has to be ramped down before you can see the poles and have your eyes try to go backward.
  • The uneven equirect projection at the poles can still give you some “extra” resolution, at least in one axis, so the bug becomes sort of a feature.

Still, it is a compromise, and some videos won’t work well in this format. An obvious example would be putting crisp text on the ceiling or floor. In exchange, we get a situation where view dependency in just the yaw angle is sufficient, which allows a system with minimal data duplication, so it can work on local devices instead of just massive servers.

“I am very excited at the prospect of unlocking quality that has laid dormant in so many high-end 360 productions made over the last several years that have only been seen at terribly degraded quality,” Carmack concludes. “Dust off those old studio masters!”

Oculus CTO Shares VR Dev Tip: 'The Formula for Avoiding Aliasing'

For developers who want to experiment with the new playback tech, Oculus has published a sample, which they’re calling the VR 5K Player, which shows developers how to build video files to work with the player, and the player itself.

This article may contain affiliate links. If you click an affiliate link and buy a product we may receive a small commission which helps support the publication. See here for more information.

  • Michael Sherman

    Great Job!! Impressive on how much better it looks

  • ArSh

    “Most people don’t look very far up or down”… er, Carmack seriously needs to look at some 5K adult VR content I think.

    • V Z

      Carmack doesn’t care about porn. The reason is that Oculus store isn’t going to sell porn and therefore there is no money for them in catering to that type of audience.

      • dk

        doesn’t care about porn? what about the Oculus Goo

        • V Z

          What about it?

          • dk

            that’s what it’s made for ….”consuming media” ….oculus goo

          • V Z

            I believe they only make money from media sold through Oculus store. Porn is obviously not sold there. I read somewhere that they don’t mean to make money through hardware sales. So, that means if you buy it for watching porn, they don’t earn anything from you.

          • dk

            let me check pornhub vr statisics after the goo was launched
            :P jk

          • V Z

            I don’t doubt that many use it to watch porn. I’m just saying that there is no money in it for Oculus, therefore they would not go out of the way to accommodate it.

          • dk

            technically a cheap standalone 3dof headset is all they need to do to accommodate it ……..also giving details about the 5k video doesn’t hurt
            ….but only technically obviously
            ….mot to mention that 3d180/360 videos is a huge part of the use case for a headset like that ….porn is just a genre

          • V Z

            My comments were specifically about Carmack implementing 5k in a way that is not very accommodating for porn. The details are in the article, which I guess you didn’t read.

          • dk

            and what I said about it is …..giving details about the 5k video doesn’t hurt…..not that his goal is better porn videos

          • V Z

            yes, it doesn’t hurt. It’s interesting to know what factors he considered when deciding how to limit bandwidth while maximizing perceived resolution.

      • brandon9271

        What if the porn director tilted the camera off axis so that the detailed area would be looking down? I know that this is generally bad practice but if you can reset the view in the video player to compensate I’m not sure what difference it would make. Another option would be to use a similar method as Carmack’s that instead concentrates on the 180° in front of the viewer and have lower res or lower frames rate behind the viewer. Even a high res, static image would probably be fine if nothing is happening behind..

        • V Z

          Don’t ask me how to make VR porn. I don’t watch it and have no idea. I was merely pointing out why Carmack doesn’t care about implementing porn friendly methods in his 5K video.

          • brandon9271

            Sorry, i hit wrong reply button :)

    • HybridEnergy

      It says that it sacrifices on the 360 video end if I’m not mistaken, I do believe most VR content people consume is 180 degrees? Which by this article would be fine?

  • MarquisDeSang

    We need Riven VR now!

    • V Z

      No. We need tomato soup now! because I’m hungry.

  • Lucidfeuer

    Great advancement for stereo cubevideo. But this is a bit overblown, now I wished there was a hybrid in between stereo cubemaps and videos, because scene frames don’t need to be buffered if not moving.

    I think that’s what Google Seurat tried to do, but since it’s Google it’s vaporware.

    • R FC

      Seurat used for Bladerunner Revelations certainly wasn’t vaporware, it was very effective within the application.

      • Lucidfeuer

        You know I found it to be so unimpressive and even badly design, that I don’t even believe they actually used Google Seurat. Or if they did, that’s just plain vaporware as it’s useless..

  • Darshan

    Cant wait to see 5K 360 Discovery Videos and Drone Panorama of famous skylines like Dubai and Singapore

  • Panoaction .

    Interesting that the original consumer One-shot system from Eyesee 360 eliminated the Zenith and Nadir points as it was clear that those areas were not a natural human action, how often do you look directly up or directly down. That being said there are cases for Zenith in certain content (Historical Buildings) and the plan for Nadir from our end has always been to project valuable data in that area. The aspect of splitting 360 images into sections for rendering go back to the 90’s wth ZOOM being the origins of it. All this of course came from GIS and mapping tech. So as much as I give credit to Mr. Carmack for his efforts and makingit happen he is only emulating what wa slready done in the past. Had Oculus loooked to the past then all this would have been in place years ago!

  • Panoaction .

    He is not alone, others actually did this back in the 90’s