It’s notoriously difficult to transmit exactly what’s happening in a VR game without actually strapping on a headset yourself. And while mixed reality capture basically addresses this by letting you see the action from the comfort of a traditional monitor, it oftentimes requires purpose-built green screen stages with external cameras—great if you have it, but it’s time, space and cost-intensive. Owlchemy Labs, the studio behind the critically acclaimed VR game Job Simulator (2016), however recently revealed an ingenious way of watching and interacting with users in VR which may be as simple as pulling the smartphone from your pocket.

Owlchemy Labs recently unveiled research on an experimental mobile app, dubbed ‘Mobile Spectator’, that uses an Android smartphone running Google’s ARCore to track the device’s position in physical space; while the VR headset user sees a floating phone, rendered within the game, the smartphone user has a window into the virtual world, letting you watch a live feed of the VR player, snap pictures, and even interact by doing basic things like tossing beach balls.

The company says in a blog post that they took a “different approach from a traditional multiplayer networking solution” to create the unique third-person view. The phone and PC communicate directly via WiFi. After an initial calibration, its AR-calculated position is sent to the PC, which then places an additional camera in the VR scene—Mobile Spectator’s point of view. The PC renders a frame from this camera, encodes the frame, sends it back to the phone, and decodes it there.


In effect, this lets you walk around the VR user’s virtual environment and freely take in the scene at any practical perspective while the VR user goes about playing the game. While it’s not mixed reality capture (example video at the bottom of the article), giving someone a window into a VR game that they can control helps address the same fundamental task of dissolving barriers between the prospective user and the VR content. As an example, there’s also implications for local asymmetric play too, as a smartphone user might see something rendered invisible to the VR player, or have to give them a key item.

Oculus Quest Supports "Arena-scale" Tracking and Multi-room Guardian

The experimental app was developed using the studio’s upcoming game Vacation Simulator; the studio says Mobile Spectator is however “a formidable undertaking,” which includes performance overhead to the PC with additional rendering and video encoding. The smartphone itself also has its work cut out for it, as it must simultaneously run ARCore and decode video, which all adds noticeable latency.

Although Owlchemy Labs hasn’t announced any official plans to include the app’s functionality in Vacation Simulator, what we’ve seen thus far is really promising.

Owlchemy Labs has been a steadfast pioneer both in object-based interactions vis-a-vis Job Simulator, but also some of the first in-engine mixed reality capture techniques which made creating the sort of reality-bending mashups a quicker and easier process.

This article may contain affiliate links. If you click an affiliate link and buy a product we may receive a small commission which helps support the publication. See here for more information.

  • Graham J ⭐️

    I imagine plenty of devs have thought of this idea but clearly a render per spectator isn’t going to scale. Better to build a mobile version of the app and have it fully independent.

    What I’d like to see is a way for the AR client to interact with the world. That’s something I haven’t seen yet.

    • Jarom Madsen

      I love the idea of a LAN mobile spectator game with native apps running on the mobile devices. Especially with the Quest coming out that already has to have it’s graphics at a mobile processor (albeit a meaty mobile processor) level, it would certainly make for quite the party.

    • Adrian Meredith

      I imagine it’s because they have a game idea they want to use it for but they’re only working on the tech right now. Really need local rendering though, would be pretty cool for cardboard/daydream users to hop in

    • dk

      “Better to build a mobile version of the app and have it fully independent.” say what ….u mean a game running on a server and various devices in the same environment participating in the same experience ….but it will take a while to get the latency where it needs to be ….and one or two spectators running with a game with simple graphics like this is not a problem

      • Graham J ⭐️

        I meant spectator devices rendering their own views, but cloud rendering of spectator views could work and would scale. As you say, latency would be the issue there but as they’re not in VR it wouldn’t matter too much.

        • dk

          as far as it being “spectator view” of a game happening in the pc ….the pc sending video is simple
          as far as a phone having a simplified game version interacting with people in vr …sure ..most likely wouldn’t be done just to be a spectator of something for a couple of minutes

  • jj

    yeah it all comes down to network programming, not too crazy tbh. it has been done before, just not by as popular of people. Don’t let my pessimistic view fool you, I love this feature and cant wait to see more of it!

  • JesuSaveSouls

    Be careful what you ar smartphone into because alien iso is no joke.Praise Jesus!

    • jj

      Do you mean the game Alien: Isolation?
      HAIL SATAN our true SAVIOR

    • gothicvillas

      666 THE LEGION 666

    • Graham J ⭐️

      Please stop posting your religious babble in technical forums. No one wants to hear it.