The 360 film industry is still young and it’s maturing rapidly. With his name on more than 20 360 film projects to date, Armando Kirwin has had an inside view as to where things started, where they’re going, and why.
Guest Article by Armando Kirwin Armando Kirwin has worked as a Director of Virtual Reality, Post Producer, Executive Producer, and Head of Post Production at companies like Here Be Dragons (formerly known as Vrse.works) and as a freelance filmmaker. He has helped create more than a dozen VR projects to date. These projects have earned one Emmy nomination, the Grand Prix at Cannes, and numerous debuts at Tribeca, SXSW, and Sundance.
There was a time in Hollywood when high-end film cameras were all proprietary and the cost of film stock per movie could be in the hundreds of thousands of dollars. Powerful incumbents enjoyed this tremendous barrier to entry until the transition to digital technology democratized the playing field.
Custom Cameras
When we started making cinematic 360 video a few years ago there was little pre-existing technology for us to use so we had to undertake the costly process of building our own.
The camera was the single biggest issue. There simply was no professional VR camera at the time and, for the companies that could afford to create one, it became a huge competitive advantage to do so. I’ve been told that some proprietary VR cameras cost as much as ten million dollars to research, develop, and tool.
But the truth is that — even with millions of dollars at our disposal — we had no business building cameras because things like optics were way beyond our ability to engineer. To this day, no true custom optics have been developed specifically for professional 360 cameras. Nor have any sensors.
And that wasn’t even the hardest part. In addition to engineering limitations, there was another major hurdle: software. VR cameras are typically made up of multiple small cameras arranged in a sphere, and after you’re done shooting you have to blend all of those cameras together into something seamless. This is known as ‘stitching’ and it was done manually by VFX artists.
The Stitching Hurdle
Stitching is an extremely labor intensive process. Imagine trying to blend 8, 17, or even 24 different cameras so that they all have the same exposure, color balance, etc. for every single frame, 30 or 60 frames per second, in 3D. In fact, because this was mostly manual labor, it cost between $8,000 and $20,000 per minute of final product (e.g. a five minute film could cost $40,000 to $100,000 just to stitch). It also took about one week per minute to complete.
Solving the stitching problem through automation required the development of complex software algorithms and once again no one, including startups with tens of millions of dollars in funding, had the expertise they needed to do right. Unfortunately, some of them pretended that they could and it ended up hurting the industry because it downplayed the true cost and difficulty of making cinematic 360 film , a trend that continues to this day. The truth is that only large companies like Facebook and Google were in a position to try, and the good news is that they stepped in to do it after about two years.
However, not even these mighty giants were able to fully solve algorithmic stitching (which is today achieved using something called optical flow). I would subjectively say that optical flow solutions are currently reaching a success rate of around 80–90%, with Google’s ‘Jump Assembler’ being the market leader. This makes sense because, generally speaking, algorithms are Google’s speciality.
A recent project that I did, which at first seems like a simple film noir series starring famous actors, is in fact a technical showcase of what you can achieve using nearly 100% optical flow stitching with minimal manual VFX. And because we had almost 20 minutes of final product, we saved our client, The New York Times, about a quarter million dollars in stitching (we also finished all nine films in only three weeks). This is why it makes sense that VR professionals are willing to spend $15,000 on the GoPro Odyssey camera (which is compatible with the Jump Assembler) or others like it.
Off-the-shelf is the New Proprietary
The GoPro Odyssey was only step one. As one of the first and only people in the world to use the best optical flow cameras on the market (including the Facebook Surround 360 and Google’s upcoming Yi HALO), I predict that optical flow-based solutions are one or two generations away from being completely reliable for most forms of production. This means that in ~18 months anyone will have access to the same caliber of technology as the very best proprietary systems held by a handful of companies today.
Even more exciting is that I am only aware of two proprietary systems in the world that are objectively better than the current generation of optical flow cameras and the quality gap isn’t even very large.
Other proprietary VR technologies were easier to solve than the camera. When we started there was actually no way to even show 360 film online once you were done shooting and stitching it. It sounds ludicrous, but it’s true. Dozens of engineers had to be hired to create custom apps. Now you can buy a white label app solution for a few thousand dollars or upload for free to YouTube and Facebook.
Nor were there any useful 360 spatial sound engines or microphones. Once again, engineers were hired and a lot of money was spent. Today there are multiple sound formats that are supported and even some proprietary systems that had a flash of success, like Two Big Ears, were purchased and made freely available by Facebook. I am only aware of two remaining spatial sound engines that are objectively better than the open solutions already on the market.
It’s now clear that early cinematic VR companies were very much like the old Hollywood incumbents. This all seems so logical in hindsight, but we ignored the obvious lesson that the means of production in Hollywood was once proprietary and eventually became completely democratized… and that it was inevitable that 360 film would quickly follow this exact same path.
As an industry we knew it was happening, but we downplayed this fact when it came to the stories we told investors, customers, and each other. At this point I would consider any company talk about “proprietary” cinematic VR production technology to be a red flag.
– – — – –
There is nothing wrong with this transition. It’s simply how production works. And, in a lot of ways, this is also 360 film reaching parity with VR games: VR game creators essentially all have access to the same tools (i.e. pick Unreal or Unity and get to work). The primary competitive advantage is creativity. And that’s the way it should be.
We should all be very excited about the new creative voices that will start to emerge because of these trends. However, the democratization of 360 film has turned the entire nascent industry upside down, and I’ll talk more about that in the next part of this series.
Read More in This Series
Disclosure: Armando has worked with Facebook on a contract basis to produce 360 film content. Facebook was also a client of Milk(vr). The New York Times was a client of Vrse.works where Armando worked on a variety of projects in various capacities including Post Producer, Producer, and Executive Producer and Head of Post Production. The New York Times was also a client of Milk(vr) where Armando was Director of VR. Armando had access to a pre-production Yi HALO camera for an unreleased project, but was not compensated by Google. Armando also had access to a prototype Zcam MFT camera, but was not compensated by Zcam.