VR software has introduced many new challenges to developers. Among these challenges, rich interactions are at the very core of all the new elements VR software designers need to consider when creating games or general applications.

Guest Article by Enrique Tromp 

Enrique is cofounder and CTO of VRMADA, a tech company providing enterprise VR solutions worldwide. With a strong passion for computer graphics and digital art, his career spans nearly 20 years in simulation, video games, and live interactive experiences. These days he loves taking on challenging VR projects and creating realistic interactions. You can follow his latest work on Twitter @entromp.

It’s no surprise that all the top selling VR games have something in common: their interactions are extremely polished and fun to play with. Games like Job Simulator (2016) paved the way of modern VR interaction while Beat Saber (2019) showed us that simple but engaging interactions can beat big studio productions in this medium as well.

Even though my background includes game development, I’ve spent most of my career developing training simulators. Currently I’m working in the enterprise VR market for training and, as you can imagine, realistic interactions here are key.

When developing a particular interaction, whether it is for games or for training, I always strive for three things: making it a really enjoyable experience, anticipate all kinds of interaction ‘misuses’, and polish it until everything feels right and then some.

Quality VR interactions are key to effective training because they are more engaging and authentic, developing new neural pathways that provide the ability to learn and perform new tasks. They work the same way good professors make lessons entertaining that keep you engaged, facilitating the assimilation of new knowledge. Bad interactions will cause frustration and decrease training efficacy.

The main challenge when developing VR interactions is the amount of freedom that the medium inherently presents to its users. In traditional videogames we use action buttons to interact with the world. The object and the context determine what will happen. In VR we use natural gestures instead; we pick things up like we would do in real-life, we manipulate complex tools, we can throw things around, all using our own hands.

This is a big paradigm shift in the way we design and implement interactions and presents a big challenge because users are inherently curious and they like freedom—especially in VR. Some will follow the ‘expected’ behaviors of the application, but others will be carried by their curiosity and inevitably test the limits of the world before them. Surprising the user by anticipating the creative ways that they interact with objects can make the experience more enjoyable and increase the sense of being in a cohesive world rather than a scripted experience.

In this article I will showcase some of the interactions I’ve developed and offer some thoughts on the design approach to each.

Lab Elements

I’ll start with a sci-fi lab that is part of a sandbox we developed internally to create and test new interaction mechanics.

The lamp on the left presents 3 handles that can be grabbed using a single hand or both hands at the same time, from the inside or the outside. It is attached to the world through a mechanical IK-driven arm that hangs from the ceiling which constrains its range of movement to a sphere and gives context for its ability to be placed in any position.

From all the changes that we made, adding haptics and smoothing out the movement of the lamp had by far the biggest impact on user experience. The smoothing filter conveys the feeling of mechanical resistance (that it’s not possible to move it around that easily) and adding haptics multiplies this feeling tenfold. It is also very gratifying to see how the mechanical arm follows the lamp around when you move it, and how it keeps swinging a little when you release it. These are little things that we add for no other reason than to keep the user happy and engaged.

The laser has a different IK setup. One of the things that we tried to experiment with is the rubber joint that joins the head with the arm and gives it two degrees of rotational freedom. We got the inspiration from the avatar wrist in the game Lone Echo and thought it was a cool way to model a ball joint.

The laser beam works by casting a ray from the tip and creating a polygon strip simulating the burn if the surface has been exposed long enough. Smoke particles help to add a bit more detail as well.

Lab Battery

This clip showcases a battery swap in the same sci-fi environment. The main purpose is to study objects with different constraints that need to be manipulated correctly in order to complete the task.

The first step is opening the door, while the object can only be rotated around its axis. Being anchored to the world has a very important consequence in the grab action: instead of the object snapping to the hand, it is the hand that will snap to the handle.

The second step is slightly more complex because it involves using both hands to unlock the mechanism that keeps the battery in place. If you try to pull out the battery without unlocking it first, the hand will maintain the grip or snap back to its position if pulled away too far.

Once the lock is open the battery can be extracted, which is accomplished by constraining the position allowing it to slide only along the tube until it comes free.

Drill

The first challenge when recreating a drill in VR is to determine the necessary conditions to start the drilling process. After all, there is no physical resistance in real-life that prevents someone from moving their hand through a wall in VR.

In this example, the conditions that are required are:

  • The drill bit must be able to penetrate the material it’s pressed against
  • The drill bit needs to be oriented at a correct angle against the surface.
  • The drill needs to be slowly pushed against the surface when the user presses the trigger.

If the user tries to pull the tool any other way than out during the drilling process, the drill will be kept in place and the hand will snap back if it is moved too far away from it.

Subtle haptics play a big role in the interaction making things such as the drill rotation or the physical resistance of the medium more believable.

Continue Reading on Page 2: Cabling and Rope & Tire »

1
2
Newsletter graphic

This article may contain affiliate links. If you click an affiliate link and buy a product we may receive a small commission which helps support the publication. More information.


  • Raphael

    Richie’s rich interactive plank drilling experience.

  • DanDei

    This illustrates very well how many parts of game design need to be reinvented for VR. I find it more and more off-putting to be put in very static environments, especially in Tutorials that limit my controls even more while telling me exactly what to do when and how. There is a natural desire to grab and manipulate a VR world in an exploratory way that is just different from monitor games.

    • impurekind

      I agree. . . .

      But one thing I actually find frustrating and often counter-productive in many current-gen VR games is them trying to do overly complicated interactions when the tech just isn’t responsive/precise enough to do them properly, particularly really sophisticated hand stuff, and sometimes I’d rather it were kept simply but reliable like it is in something like Job Simulator for example (which still has one of the most interactive environments in VR to date).

      Precision, reliability and fun is more important than just trying to be more simulationy of everything even when it’s often rather clunky and clumsy.

      • DanDei

        That is why I am very, very much looking forward to Boneworks. They really seem to have nailed the physics and object interactions without being clunky.

        • impurekind

          Well it doesn’t look perfect but it’s definitely some of the best realistic interaction in VR I’ve seen.

          • asshat

            yeah at first they were kinda overly zealous about their videos, like most of thier physical interactions have been done by other before. but yeah they seem to be spending a lot of time making sure they work well. if they dont after all this itll make them look bad as well

      • A really good point. I decided to not add finger usage with the Index Controllers for this very reason. It wasn’t as precise and it just led to having to explain more on how to use the feature. I stuck with a ‘pincher’ like movement that was activated by gripping the center palm shaft. I also allowed a larger sweet spot for grabbing to reduce frustration in picking up smaller rocks and tools. Not as realistic but it seemed to be just as satisfying for the novice who may have never used a VR controller. I also think the Valve Index controllers add much needed transparency to allowing for more realistic hand actions.

        I also what to give a shout out to Valve for providing their instructional demo that has been my inspiration in interactivity and setting an early standard on what VR should be. In saying that, I still feel there are way too many buttons that get in the way of experiences that aren’t always intuitive or consistent. Sadly I ran into this as well and started replacing buttons with actual hand movement or ‘laser’ pointers using just the triggers.

    • PJ

      Agree, this is exactly why Skyrim and Fallout didn’t translate well into VR for me, ok it’s great to be in those worlds, but what’s the point if as soon as you want to intersect with it, you can’t

  • Ratm

    Lol NO. Key is to stimulate the fantasy not put the user to work,this gets boring fast .Beatsaber has 1000 songs and all patterns are different.
    Job simulator was in the wow! Vr era its literally trash not a game.

    • Raphael

      I find beatsaber mindnumbingly boring. Slashing shapes is a very tedious game mechanic. I know it’s popular but that’s mainly because it’s accessible without any brain power to non-gamers.

      • Peyton Lind

        I’ve owned Beat Saber for about 7 weeks on my Quest with custom songs installed the entire time. While I am not yet there in finding it mind-numbingly boring, I am getting a bit bored of it and agree with what you said about one reason it is so popular.

        • Raphael

          It’s a good game in the same way Tetris is a good game but it’s the dietary equivalent of McDonalds. I can’t exist on a diet of McDonalds so games like that are short-burst for me. For developing fast reflexes BS is unbeatable though.

          For me a healthy diet is DCS World, Elite Dangerous, No Man’s Sky and anything else with depth and longevity.

          For exercise in VR I mainly use BoxVR (wearing 1.5kg wrist weights).

          I haven’t tried Quest yet but looks like it has a bright future.

          • Marcus

            It’s the other way round: For most people sitting is extremely unhealthy while moving (like in Beat Saber, Ratchet NX etc.) is so much more healthy. It may only be a metaphor of yours, but I don’t like it.

          • I finally got a chance to try the Quest and was pleasantly surprised. First, I did not know that they created projected black & white stereo paired image back to the viewer, so I thought I was in some office type VR experience, then I realized it was my own space. Once the familiar Oculus landing page came up after identifying my boundaries it was less confusing, (but still the thought it was cool). After that I tried “Robo Recall” to see how much it strayed from the desktop version and delightfully it seemed to be just as engaging. Yes the graphics take a hit, but not as much as some reviews seem to think. I for one found the experience quite convincing, and in fact more liberating since I didn’t have to worry about where I was, or have my position abruptly affect my view due to turning and having my headset lose tracking since I only have the two camera setup.

            The headset is comfortable but noticeably heavier than the Oculus Rift CV. Image quality is better and audio seemed about the same, but not doing a side by side comparison, I cannot say for sure.

            My only complaint has more to do with the business strategy of Facebook. By pushing curated content and (expensive) apps in their store, trying to turn it into virtual social activity and hampering connectivity with the desktop, I believe they will kill the market just like Google did with Daydream. It would have been much better headset if they built in a WiGIg wireless solution to turn it into the Vive killer. From my understanding in the research of the SLAM data, it is processed locally on a custom (Qualcomm) chip which would reduce dramatically the bandwidth needed (MS WMR) since the camera streams don’t need be streamed to the host system. If this is true, which I believe to be due to all the third market solutions using WiFI, this would have negated having to bring two systems to the market. But here I believe like many companies before it, we are seeing what happens when two camps are set up within a company pursuing different business models.

      • That’s the key Raphael – “non-gamers.” But as I found in my “Lunar Excursion VR Experience.” Creating actions that come easy to the novice is what draws them in. But it is also external cues, like wearing “moon boots,” and being assisted by those dressed in white lab coats and coveralls with appropriate patches also helps the patron/novice believe in the experience. But the “assistant” plays a vital role and ensuring the novice makes a smooth transition into the virtual world. I think this is why the Void has been so successful because your experience does not start once you put the headset on.

        Also back to Beat Saber. I think something else they did is make the experience for those not participating in VR engaged while watching the monitor. But the same could be said for a number of console type games which put a lot of effort in making the experience enjoyable to those spectating. This is something I tried to accomplish as well by recreating a black and white camera to simulate the B/W Westinghouse that beamed back images we saw on TV with of course the new “third (player) astronaut” introduced to the scene (there is a small color inset view of what the user is experiencing as well). This helps the spectator relate to what the patron/player is experiencing, since many of them blurted out “Did you see that?” to their friends.

    • Peyton Lind

      All patterns are not different. WTF are you smoking?

      I think stuff shown here looks far more interesting than what I do in Beat Saber.

      • Ratm

        So.. You beat all songs with the same repeating moves?Idd my smoke is better than yours !!

    • Żéñ Źdźbło

      Can’t believe we’re comparing beat saber to this.

  • namekuseijin

    rich interactions is key indeed in VR. but I hope that doesn’t mean 90% rich interaction with a gun in your hands and 10% rich interactions with a few puzzles to pace the longevity of the game…

    all in all, it’ll still take some time to see VR evolving to integrate better physics so rich interactions become more prevalent… more grabbing and handling things with your own hands and less magnetism… inventory management is an area that should improve much from better systems with rich interactions. this is an area in which native VR titles really shine even now compared to flat ports…

    • See my comment and couldn’t agree more. There are so many world experiences that many of us will never experience (or want to) and constantly seeing experiences that simulate killing, maiming or warfare may do more to keep VR out of the reaches of the public than anything else.

      It should be noted that it was Myst, The Sims, and SimCity, Madden Football, MS Flight Simulator that brought computer gaming to the masses. And frankly we should not forget that some of most popular apps in VR have nothing to do with holding a gun. On that note I have to say my guilty pleasure has been Arizona Sunshine but feel their inventory system could be improved, but then again when I wear my camera vest in doing a medai shoot, I always forget which pocket I put something in and always doing the “pocket dance.”

  • I love these articles about UX! I miss the ones by Leap Motion…

  • Sandeep Banerjee

    Exactly right. When I play VR, I want to do stuff that is *impossible* in flat-screen gaming.

  • LiquidKaos

    Thanks for the great article. It was really informative and presented in the perfect way to keep me truly engaged.

  • MatBrady

    VR Studios don’t know what to do with VR, and this article is case in point. Tasks are just not fun for most people. Look at the end scene in Lone Echo as a terrific example. They had a high-quality assets, a terrific locomotion system, great characters, and what was the big finale?? Menial tasks with a time limit. Are you freaking kidding me? Stormland looks like it’s going to get it right, but the kinds of things demonstrated in this article screams lack of imagination. We’re going to look back at this kind of stuff in two years time and groan.

  • Excellent article and couldn’t agree more as I have found in my own testing of “Apollo 11: ‘One Small Step For…’ VAMR Experiences,” “Lunar Excursion VR Experience.” Interaction is what drew patrons to the event on 7/20/2019. Not only did the patron use their hands, but also their chest and feet since they actually donned a (virtual) A7L lunar excursion astronaut suit through the use of three Vive Trackers. This not only allowed them to see their limbs, but also experience what it felt like to be encapsulated. This became apparent in not only in my own testing, but watching others try to look up and realize since the helmet is not connected to their head but to their shoulders, they had to lean back to look up and see the earth or Neil Armstrong climbing up the ladder at the end of their excursion/experience. I also provided a removable chest attached Hasselblad 500EL camera so the user can capture their own images that are time stamped and saved in a folder on their harddrive. Again because it is attached to their chest, they have to bend at the waist, or squat to capture images. Finally, one of the more important things I wanted to convey is how 1/6 the gravity of earth would affect objects, so many of the tools and rocks can be picked up or kicked around, which as you can imagine do not react the same way they would on earth due to the limited gravity and not atmosphere. All of these things led to a very convincing experience that brought some of the older participants to tears, since like this old timer, watched the event on TV in 1969 (I was eight).

    If you are interested in trying it out yourself, it is my goal to release it in September on Steam, Vive Portal and Microsoft Store for FREE after I further optimize the build size and add a version to those limited to an HMD & controllers, with an extra “sandbox” option that allows the user to roam the area around the LM using teleporting and “Kangaroo Hopping” mode for those who don’t mind roller coaster type movement through the measuring of the controllers vector acceleration (pushing down) to control jump force and angle.

    Finally, after its release I will be introducing “Ascent” a simulation of the Apollo 11 LM liftoff from the lunar surface and docking with the SCM (Service Control Module) that uses another technique I am working on and that matches physical controls to their virtual versions using proprioception. In this case several buttons, an analog thrust control and 8-way joystick for RCS (Reaction Control System) to allow you the pilot to lift off, reach orbit and navigate and dock with the SCM. The goal is to create a flexible kosk like system with ‘lego’ like components attached to a perforated surface to match the controls to the VR equivalent for all kinds of vehicle and control system simulations. More to come in the next couple of months.

    Below seen in action with Joshua Young, facilitator of the ‘Reality Design’ Meetup in Portland, OR. Images are from the virtual Hasselblad 500EL attached to my chest 1Kx1K with Lareau plate markings.

    https://youtu.be/uMPBtMTI55g
    https://uploads.disquscdn.com/images/e43eb8e3f32fae1731f176f7b3b534c3a8611bfa12cc87a00f2cb751c87d9e38.gif
    https://uploads.disquscdn.com/images/59d7251a5e7ec0ee6e76629e1ecf5e16e58e8e6e849e1050c897ab2942ec8699.jpg
    https://uploads.disquscdn.com/images/6d80f3a24361b5a11473392cfec8dc3728224900d277b7fb4d99a4d757e93d3c.png https://uploads.disquscdn.com/images/b25d5efa83664b987ab6be88df39d4e7c576a87d372027caf08a3da26e5ffc32.png https://uploads.disquscdn.com/images/6e62ff80ca4c5b757250cc2a73267d3d21ca2534587c87333c485e75aa5cb874.png
    https://uploads.disquscdn.com/images/c6e9926ca7a9ef1cfa85f3056cbcc911a5ba39906b53d60b7c73d9135105c075.png