VFX professional Kert Gartner has been pioneering new ways to make capturing virtual worlds as compelling as filming in the real world. His latest technique makes a real virtual camera out of an extra Vive controller and a smartphone.

You may have actually seen Gartners work before, he was a major part of the new Space Pirate Trainer trailer production, and the mixed reality Fantastic Contraption trailer. His latest trick is a little bit of genius that expands upon much of his prior work with VR cinematography.

To get the feel of ‘real world’ cinematography in VR, you need to be able to manipulate the virtual camera just like real world cinematographers manipulate real cameras. Often times that means handheld cameras that are up close to the action. Gartner has been emulating that handheld camera look by smartly using a third Vive controller to manipulate a virtual camera inside the virtual world. But filming this way is difficult because the controller doesn’t have a viewfinder, meaning that seeing what it’s pointed at requires looking at some distant monitor on the periphery of the of the action.

Now Gartner has solved that problem by using a smartphone to act as the viewfinder for the virtual camera, allowing him to see exactly what the virtual camera sees as he films:

The foundation of the technique requires a custom build of whatever VR experience is being filmed; one which is able to project multiple camera views in a single frame, including an angle which will be the view of a virtual camera which is “attached” to an extra Vive controller. Rendering out a 4k frame divided into multiple views means that each view can still be captured at a sharp 1920×1080 by cropping away the other views. Capturing all views at once gives the editor the ability to chose from multiple viewpoints of the same shot in post processing, without the cinematographer needing to do multiple takes from different angles.

Former Codemasters CEO Joins nDreams Board of Directors as Chair

virtual-reality-camera-vr-camera-htc-vive-1The next step is to affix a smartphone to the Vive controller. In this case, Gartner whipped up a simple rig with a vertical grip for easy capturing, but I imagine it would be almost as easy to connect the smartphone directly to the Vive controller (just don’t block the tracking receivers!).

From there, Gartner employs the jsmpeg-vnc utility which allows him to stream a selectable crop of the monitor to a web server which allows it to be viewed easily by visiting a URL on the smartphone. This gives him a viewfinder which shows exactly what the virtual camera sees.

virtual-reality-camera-vr-camera-htc-vive-4Altogether, the rig now works like a handheld camera, except instead of filming the real world, his viewfinder shows a portal into the virtual reality world. Even rendering multiple views at 4k and running the screen streaming and capturing software, Gartner says the whole system is still hitting the crucial 90 FPS figure (no doubt thanks to a powerful PC managing it all).

Praise to Gartner indeed for this smart solution, though many will know that technique isn’t brand new; the use of real-time digital visuals with a virtual camera and viewfinder has been seen in high-end film and game production, notably on Avatar (2009), Halo Reach (2010), and Uncharted 4 (2016) but Gartner’s system is tremendously less expensive, using just a PC, HTC Vive, extra Vive controller, and some streaming software; that puts it in the realm of practicality for many VR developers.

We expect to see this technique further developed for the production of high-end VR trailers, and even for big livestreaming VR shoots. Hooking the Vive controller and smartphone viewfinder up to a Steadicam rig could give a cool high-end film look. We can also imagine a more realistic virtual camera model offering complex adjustments like focal length, exposure, and manipulation of lens characteristics, all controllable in real time from the touchscreen of the smartphone viewfinder.

This article may contain affiliate links. If you click an affiliate link and buy a product we may receive a small commission which helps support the publication. See here for more information.

  • MW

    Great idea and great work. Sadly, it’s
    just idea for now, because we have to wait (maybe many years) for hmd with good enough screen and real vr fov for real hi-end vr movies:-(

    • dogtato

      This is for filming in VR, not for viewing in VR. The end result will be a traditional rectangular video.

      • Theo

        could be quite cool to create some scenes and then allow a camera man to just film the action from where ever as they walk around. only problem is the limitations of real time i guess.

  • Sponge Bob

    trying to understand the utility of this….

    virtual filming inside virtual world

    like subsampling digital data for virtual world to produce some type of limited dynamic view into that virtual world

    I say – who cares ?

    and why smartphone ? it’s all computed predetermined digital data anyway so you can manipulate it and view any way you want on any device – in virtual world or in real

    • pj530ii

      It’s not useful for players but will be great for people trying to create “mixed reality” content. Aside from actually trying the games, mixed reality is currently the best way to show off a VR game. Having the camera man be able to see what’s actually happening in the game world (as opposed to just looking at a person flailing around in front of a green screen) will allow them to create better videos/trailers to show off games.

  • Philip Anderson