YouTube Spaces Studio Explores the Basics of Mixed Reality Video Capture

8

Prolific YouTuber Tom Scott recently visited the Mixed Reality Lab at the YouTube Space studio in New York in order to showcase mixed reality capture. The lab is using a technique involving a green screen and a tracked camera to combine real world footage with a virtual reality environment, and helpfully lays out the basic concepts of the technique.

One of the persistent hurdles when explaining or promoting augmented and virtual reality technology is the need to convey the experience when viewed on a conventional display. One effective way to achieve this is with mixed reality, a combination of capture techniques that merge footage of the real world with the virtual world.

Northway Games, developers of Fantastic Contraption, were among the first to embrace the technique, and their various guides are a good place to start for those already versed in digital capture and streaming. Valve’s introduction video to SteamVR at the launch of the HTC Vive last year remains a superb illustration of the impact of presenting VR in this way.

YouTube Space New York, one of several official learning and creative YouTube studios around the world, has a Mixed Reality lab using an HTC Vive setup, which received mainstream exposure on late night talk show Conan in November. Recently Tom Scott, a popular YouTuber, visited the lab, providing a somewhat more sensible presentation about the equipment as part of his ‘Amazing Places’ series. The video is seen heading this article.

vive-tracker-and-accessories-6
A Vive Tracker attached to a camera lets the computer sync the virtual and real footage together spatially.

Tom Small, manager of New Technology Programs at YouTube Spaces and Travis Butler, manager of Technology at YouTube Spaces in The Americas were on hand to explain some of the details. Small describes how a third Vive controller attached to the handheld camera filming the green screen allows the software to create a virtual camera that tracks in the same way, and then both types of capture can be combined to create the mixed reality footage. The upcoming Vive Tracker is ideal for this application, rather than having to use a third controller.

Butler explains the need for a powerful PC to achieve best results; in this case they’re using a high-end Intel CPU, 32GB RAM, and an NVIDIA GTX 1080 graphics card in order to render at 4K. The system is capable of generating the composite image in realtime, and a higher quality output can be created with post processing. The high system requirements are also due to the additional views being rendered alongside the normal VR output; the virtual camera view and a foreground and background layer used to provide a basic depth effect.

SEE ALSO
Smart Solution Makes Handheld VR Cinematography Easy with a Real Virtual Camera

This ‘video sandwich’ method is achieved by rendering a view of the ‘background’ and ‘foreground’ separately, determined by the position of the headset relative to the virtual camera. Any geometry that appears between the headset and the camera is considered foreground, and anything ‘behind’ the headset is background.

It’s a demanding and inaccurate solution, and a problem that Owlchemy Labs, developers of Job Simulator, have addressed using a more advanced approach. Their first blog entry explains the difference between the common mixed reality solutions of green screen overlay and the foreground-background sandwich method, and why their depth-sensing solution is so much better; a custom shader and plugin uses depth information from a stereo camera to render the user in-engine in realtime, meaning no external compositing is required, and accurate depth is achieved. In addition, their second blog entry shows how this depth information means the user can receive dynamic lighting from the game engine in realtime, as well as solving transparency problems and removing green screen boundaries.

This article may contain affiliate links. If you click an affiliate link and buy a product we may receive a small commission which helps support the publication. See here for more information.


  • Brandon Smith

    I find this fascinating. I would ABSOLUTELY watch a reality show that was composed of nothing but VR gamers playing a game that was real to them and television to us.

    • NooYawker

      I would imagine Twitch is already planning on it. If not, they better get on it.

      • Brandon Smith

        I think it’s the kind of thing that would need sponsorships. I don’t know if there is really any for-sale version of the tech yet. It think only Valve has it. I should really look into it.

        Since the dawn of reality shows, I’ve always been fascinated by the idea that if you take a bunch of people and lock them up together and apply bizarre rules, they will begin to accept those rules and develop a society based around it.

        For a while, I’ve thought it would be really cool if they did a gamer based reality show where they take a bunch of random people, lock them up in a set of a space station and have them be pilots in a sci-fi drama. Like Battlestar Galactica, basically. But if the siren starts blaring at 0300 hours and they’re all asleep, they would all have to get up and go running to their ships Wing Commander style and fly a mission. The cockpits could all be real sets, but once they ships launch, it switches to virtual reality. The old show Space Above And Beyond had a system where the cockpits were inserted into the ships seperately, so the show could do something similar so they wouldn’t have to build actual planes. But the kicker would be that if you die in a mission in the show… you just don’t come back to the station afterwards. You’re off the show. And the reality show contestants would just have to deal with you being gone. Maybe reinforcements are added periodically. But the overall plot of the show could be like EVE Online and be a world that any person watching the show could participate in MMO style. So the real world events of the game would affect the show and the show would affect the game.

        I would definitely watch something like that.

      • Brandon Smith

        I think it’s the kind of thing that would need sponsorships. I don’t know if there is really any for-sale version of the tech yet. It think only Valve has it. I should really look into it.

        Since the dawn of reality shows, I’ve always been fascinated by the idea that if you take a bunch of people and lock them up together and apply bizarre rules, they will begin to accept those rules and develop a society based around it.

        For a while, I’ve thought it would be really cool if they did a gamer based reality show where they take a bunch of random people, lock them up in a set of a space station and have them be pilots in a sci-fi drama. Like Battlestar Galactica, basically. But if the siren starts blaring at 0300 hours and they’re all asleep, they would all have to get up and go running to their ships Wing Commander style and fly a mission. The cockpits could all be real sets, but once they ships launch, it switches to virtual reality. The old show Space Above And Beyond had a system where the cockpits were inserted into the ships seperately, so the show could do something similar so they wouldn’t have to build actual planes. But the kicker would be that if you die in a mission in the show… you just don’t come back to the station afterwards. You’re off the show. And the reality show contestants would just have to deal with you being gone. Maybe reinforcements are added periodically. But the overall plot of the show could be like EVE Online and be a world that any person watching the show could participate in MMO style. So the real world events of the game would affect the show and the show would affect the game.

        I would definitely watch something like that.

        • NooYawker

          You have given this a lot of thought. I’m not sure if I would watch this, I hate reality shows, but I like competitive shows like Top Chef. This could be Top Wing Gunner or something. Maybe I’d watch it… I’ll probably watch it. :D
          Go sell it, to bravo, or aim low and go to TLC. You’ll never know!!!!

          • Brandon Smith

            I hate reality shows when they are based on bringing out the worst in people. But I love reality shows when they are about putting real people in fantastic situations. There was one reality show that came on during the reality show boom of the early 2000s called Murder In Smalltown X that was AMAZING. The whole premise was that it was a murder mystery event that took place in an entire town where every single person in town was a paid improvisational actor. It had a Twin Peaks/Lost feel to it. At the end of every episode getting “voted off the show” meant that you had to go investigate a site for clues by yourself and one person would get “killed” by the murderer and not come back. It was SUCH a cool idea.

          • Brandon Smith

            I hate reality shows when they are based on bringing out the worst in people. But I love reality shows when they are about putting real people in fantastic situations. There was one reality show that came on during the reality show boom of the early 2000s called Murder In Smalltown X that was AMAZING. The whole premise was that it was a murder mystery event that took place in an entire town where every single person in town was a paid improvisational actor. It had a Twin Peaks/Lost feel to it. At the end of every episode getting “voted off the show” meant that you had to go investigate a site for clues by yourself and one person would get “killed” by the murderer and not come back. It was SUCH a cool idea.

        • NooYawker

          You have given this a lot of thought. I’m not sure if I would watch this, I hate reality shows, but I like competitive shows like Top Chef. This could be Top Wing Gunner or something. Maybe I’d watch it… I’ll probably watch it. :D
          Go sell it, to bravo, or aim low and go to TLC. You’ll never know!!!!