Jules Urbach, CEO of OTOY, believes lightfields are the endgame of virtual reality. Essentially a way to digitally record all light reflected from a real or virtual object, lightfields seemingly enable feats of technological magic, from being able to view near-photoreal quality scenes from any angle within the capture volume to actually being able to dynamically relight actors in real time. OTOY believes that lightfields solve many of VR’s rendering challenges and has been investing itself in them heavily. Road to VR guest reporter Nate Kozak spoke with Urbach about the company’s new lightfield streaming.
Of course, lightfields bring with them a host of challenges. Chief among them is simply file sizes; gathering and storing detailed lighting information about even one captured actor or object involves enough data that more complex scenes quickly become unrealistic to deliver across most residential connections. A few gigabytes for a movie is manageable; a few terabytes would not be. Had holographic data storage ever filtered down to the consumer level, we might be buying lightfields from catalogs, complete with Ziploc packaging, but today data consumption is increasingly across wireless links, a step in the wrong direction for a technology that loves to suck up storage space.
Luckily, OTOY has been working hard on this problem, and says they’ve finally cracked it. Sophisticated compression algorithms manage to tame lightfields, and clever tricks “sort of like foveated rendering” allow OTOY’s lightfield player to only request sections it predicts the viewer might look at next, the company says. The proof of concept culminates, stunningly, in a transmission rate of 1.5Mb/s for a single actor lightfield, far less than watching a Netflix stream, and easily streamable to a smartphone.
[gfycat data_id=”GrossHarshFossa” data_autoplay=true data_controls=false]
The demo itself was not as flashy as the worlds of opportunity it could represent. Running in AR form on a smartphone with a fiducial marker for tracking, a human actor dressed up as Green Goblin (from Spiderman) was seen on the screen. At any instant, the image of his psychotically grinning face was as clear as a high-quality real life photograph, teeth glistening and hints of transluminescence made it clear this was no mere 3D asset.
Even at distances threatening to clip into the capture volume, every detail was retained, the definition of photorealistic. Photos, unfortunately, don’t allow their viewers to reframe the image and look at the back of a head in a portrait, but lightfields do. Passers-by surely assumed this was just someone waving an expertly crafted 3D model around, because many have not grasped what makes lightfields different, much less what they could mean for the future of immersive media.
Everyone may start to get an inkling very soon though as OTOY tweeted just prior to the start of Connect that lightfield streaming will come to Gear VR in the next ORBX Media Player app update.