Making the Virtual Real World More Real

Since the lighting of the scene is ‘baked’ into the photo-based textures, the raw capture won’t include volumetric light or specular lighting. However, the team has shown how they can layer in some of these effects to make the scenes even more realistic.

realities photogrammatry virtual reality (1)
This is an in-engine screenshot, not a photograph.

In another space that I saw—an abandoned room with an old grand piano at the center—the team had added volumetric light rays streaming through the high windows along the side of the room. Following the beams from the window to their landing point on the opposite wall, they reconnected with the baked lightning from the real scene in a convincing way. With the rays illuminating the (added) dust particles floating about, the decrepit space was yet more convincing.

microsoft hololens holographic video technique (3)
See Also: Microsoft Details HoloLens Streamable Live Action Video Recording Technique

Another scene I got to see was an abandoned shower and bathing space from an old hospital. Ceramic tiles covered the space from floor to vaulted ceiling. Instead of the walls growing flatter and less real as I approached (as you might expect in a videogame), getting closer actually revealed them to be even more real than they looked from afar, thanks to the team’s photogrammetry process which was accurate enough to capture the mere millimeters of elevation change between the tiles themselves and the sunken grout lines between them.

'Puzzling Places' is a 3D Jigsaw Puzzle for Quest Featuring Stunning Photogrammetry Scans

htc vive realities photogammatry virtual reality (1)

The realism of the tiles is further enhanced because the Realities team added in-engine specular lightning, showing reflective highlights on the semi-reflective tiles which moves appropriately as you move your head throughout the scene, thus eliminating the baked lightning effect. This is especially impressive up close and at extreme angles where usual techniques used in many non-VR experiences (like bump-mapping) would break down.

Go to Places You Can’t Go

So Realities can capture impressive looking real-world spaces and bring them into VR. Now what?

realities photogrammatry virtual reality (3)
This is an in-engine screenshot, not a photograph.

The company’s current approach is to capture amazing places around the world and allow people to visit as virtual tourists, with a bit of an educational bent. Some of the early user interactions within these spaces involves finding photos positioned throughout the area which, when retrieved, reveal interesting information about the space. The team especially wants to capture places people aren’t normally allowed to go. For instance, all of the scenes I described in the abandoned building were of a place the public isn’t actually allowed.

It’s a compelling idea, especially given the rate at which the team’s photogrammetry technique is improving. I imagine it would be awesome to step inside the White House’s Oval Office, but even if I was allowed to do so, it might not be compelling enough for me personally to warrant a trip to Washington D.C. But it’s almost certainly compelling enough to warrant merely putting on a VR headset if it means I can see a near-identical recreation. And there’s no doubt that the recreated scenes would be accurate enough to almost entirely replicate the experience of walking around an art museum.

'Puzzling Places' is a 3D Jigsaw Puzzle for Quest Featuring Stunning Photogrammetry Scans
realities photogrammatry virtual reality (2)
This is an in-engine screenshot, not a photograph.

As a gamer at heart however, I was definitely thinking of ways these spaces might be used for more-action oriented interactive experiences. With the team already rendering these scenes in real-time in UE4—even with VR’s high end processing demands—it seems like that should be possible. Anyone up for team deathmatch in the Oval Office?

Realities is planning to launch a version of their photogrammetry captures on SteamVR for free. Timing on that release is not yet determined, but the company will be showcasing their work at GDC this coming week at the Graphine booth (Main Hall, booth 429).


This article may contain affiliate links. If you click an affiliate link and buy a product we may receive a small commission which helps support the publication. See here for more information.

  • Dr_I_Needtob_Athe

    Man, it’s pure torture waiting for my HTC Vive.

    • Muddy

      Hell yes!

    • realtrisk

      Couldn’t agree more! Time hasn’t gone this slow for me since I was a kid waiting for Christmas!

    • Peter

      – New pc; check
      – Room cleared: check.
      – Several days off from work after release; check.

      lets…lets browse the web some more for vive news and youtube clips…

  • Haczar Criollo

    Can’t wait for hidden object games that use this tech. Great job by the Realities team

    • Some Guy

      Perfect timing for Easter…..Lets do some VR Easter Egg Hunting

  • Jeremy Swanson

    I gotta wonder what this tech could do for crime scenes. How helpful would it be for a homicide detective to be able to go back to the pristine crime scene in VR whenever needed?

    • kr_metal

      The only issue would be virtual tampering. Less than honest workers may tamper with the virtual crime scene to frame or divert attention away from a suspect. There probably will be a law that when used for law enforcement, they have to use untampered info.

      • boniek

        While certainly possible digital signature of scene files could prevent this after creation.

        • brandon9271

          Exactly! They could just use something basic like MD5 checksum and put the file on a disc inside of a safe… but honestly it not like crooked cops couldn’t tamper with REAL evidence already. I don’t see this as an issue.

      • Jeremy Swanson

        How does this risk differ significantly from the photos they currently take?

      • Median N. Mean

        Police still use dioramas to train investigators. This would oefinitely be an improvement.

      • Infiniteworld

        Have you read The Great North Road?
        Highly recommend :)

    • Rufus

      I watched a short video series on YouTube about this very thing. They used a drone to fly over a crime scene and photograph everything so they could create a 3D model and analyze it.

  • Kai2591

    Simply awesome :)
    I started to imagine a fantasy mmorpg with this kind of REAL life graphics.
    That would be either extremely cool or extremely bizarre/surreal (uncanny valleys everywheree~)

  • Tako

    Pretty sure the devs joked about outsourcing the ‘lighting’ engine, and not the ‘lightning’ engine, although the latter would be pretty awesome too…

  • Oliver Markowski

    Since when is this technology so special? Everybody is using it since ages and Agisoft Photoscan is an absolutely amazing and fully featured tool to do that kind of stuff.

  • marko

    Getting the photogrammetry in the computer is one thing.

    Getting it out is another.

    I saw a Euclideon demonstration at the World Science Festival in Brisbane last week which shows real-time ‘unlimited detail’ virtual reality rendering of 3D photogrammetry captures.

    • Infiniteworld

      I thought those guys were totally vapourware. Are they really making stuff? Their demo videos from a few years ago logged amazing

  • Simon Che de Boer our example. We use normals and dynamic lighting.

  • PaulDean

    Please excuse me and feel free to have a laugh or 2 at my expense but I
    am only an egg. I have never been a gamer but am VERY INTERESTED in
    acquiring your Inside Realities program for my HTC Vive VR. I am also
    looking for educational titles as it were as in Museum Tours, Ocean and
    Space Exploration, etc. but am more interested in the Inside Realities.
    Can you tell me where I might get it? Any/all help is GREATLY

  • Scott Barrett

    I would love a death match in the oval office, providing Trump could be an executable NPC. Um… did I just get myself put on some FBI list?

  • Very nice one. Second best after amazing Auckland VR experience “MANA VR”. Our team finished now a very real virtual reality development on Android in which you can clearly walk (no flashing or switching!) by using your bluetooth controller and if you don´t have that, you can simply enjoy the guided tour. Find your proper Samsung S7 download here: